Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filebeat modules: Machine learning jobs #4506

Merged
merged 1 commit into from Jun 23, 2017

Conversation

Projects
None yet
4 participants
@tsg
Copy link
Collaborator

commented Jun 15, 2017

This adds support for loading ML configurations (job + datafeed) from the filebeat modules. An example ML configuration is added to the Nginx Filebeat module. This sample applies ML anomaly detection on the response codes.

The loading is implemented as part of the setup command and part of the --setup flag.

If a job configuration with the same ID exists, it is not overwritten, because deleting jobs could potentially delete user data. The user should manually delete the jobs in the UI if they want to upgrade.

ToDOs for the overall implementation:

  • Implement the jobs loading support in libbeat
  • Read the jobs configurations from the filesets
  • Add xpack to default testing env (snaphost.yml)
  • system tests
  • Docs & changelog
@tsg

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 20, 2017

I pushed everything, but I expect the test to be failing until the Kibana snapshots are fixed (there's an issue with the current ones). But should be otherwise ready for reviews.

@tsg tsg removed the in progress label Jun 20, 2017

@tsg

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 20, 2017

Also, I'm leaving docs for later, since this will mainly mean document the setup command.

@exekias
Copy link
Member

left a comment

LGTM in general, but some tests are failing, looks legit, you may want to take a look to them

@@ -66,8 +67,14 @@ func New(b *beat.Beat, rawConfig *common.Config) (beat.Beater, error) {
}
}

// In the setup command,

This comment has been minimized.

Copy link
@exekias

exekias Jun 22, 2017

Member

spurious comment?

@tsg tsg force-pushed the tsg:machine_learning_jobs branch 2 times, most recently from 0b9286e to 3c04b7f Jun 22, 2017

Filebeat modules: Machine Learning jobs
This adds support for loading ML configurations (job + datafeed) from the filebeat modules.
An example ML configuration is added to the Nginx Filebeat module. This sample applies
ML anomaly detection on the response codes.

The loading is implemented as part of the `setup` command and part of the `--setup` flag.

If a job configuration with the same ID exists, it is not overwritten, because deleting jobs
could potentially delete user data. The user should manually delete the jobs in the UI if they
want to upgrade.

@tsg tsg force-pushed the tsg:machine_learning_jobs branch from 3c04b7f to 68641f6 Jun 23, 2017

@tsg

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 23, 2017

Squashed and rebased. Remaining test failures seem unrelated, but I'm hoping for some more green on this run.

@exekias

This comment has been minimized.

Copy link
Member

commented Jun 23, 2017

💯 tested by hand, it adds the job both with filebeat setup and filebeat run -setup. Just one note, the job is not started after adding it, so the user still has to start the job by herself

@exekias exekias merged commit 1213483 into elastic:master Jun 23, 2017

6 checks passed

CLA Commit author is a member of Elasticsearch
Details
beats-ci Build finished.
Details
codecov/patch 32% of diff hit (within 100% threshold of 63.42%)
Details
codecov/project 63.35% (-0.07%) compared to 7059183
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@@ -0,0 +1,44 @@
{
"job_id": "JOB_ID",

This comment has been minimized.

Copy link
@ruflin

ruflin Jun 27, 2017

Collaborator

Only skimmed through the PR but this kind of looked fishy?

This comment has been minimized.

Copy link
@tsg

tsg Jun 27, 2017

Author Collaborator

Yeah, it gets replaced in code with a generated job id. Didn't know how to reflect this otherwise.

This comment has been minimized.

Copy link
@tsg

tsg Jun 27, 2017

Author Collaborator

To clarify, I don't mean a textual replacement. The json gets decoded and the key replaced.

@tsg tsg added the needs_docs label Jun 30, 2017

tsg added a commit to tsg/beats that referenced this pull request Jun 30, 2017

Filebeat modules: Machine Learning jobs (elastic#4506)
This adds support for loading ML configurations (job + datafeed) from the filebeat modules.
An example ML configuration is added to the Nginx Filebeat module. This sample applies
ML anomaly detection on the response codes.

The loading is implemented as part of the `setup` command and part of the `--setup` flag.

If a job configuration with the same ID exists, it is not overwritten, because deleting jobs
could potentially delete user data. The user should manually delete the jobs in the UI if they
want to upgrade.
(cherry picked from commit 1213483)

@tsg tsg removed the needs_backport label Jun 30, 2017

tsg added a commit to tsg/beats that referenced this pull request Jun 30, 2017

Filebeat modules: Machine Learning jobs (elastic#4506)
This adds support for loading ML configurations (job + datafeed) from the filebeat modules.
An example ML configuration is added to the Nginx Filebeat module. This sample applies
ML anomaly detection on the response codes.

The loading is implemented as part of the `setup` command and part of the `--setup` flag.

If a job configuration with the same ID exists, it is not overwritten, because deleting jobs
could potentially delete user data. The user should manually delete the jobs in the UI if they
want to upgrade.
(cherry picked from commit 1213483)

monicasarbu added a commit that referenced this pull request Jul 3, 2017

Cherry-pick #4506 to 5.x: Filebeat modules: Machine learning jobs (#4588
)

* Filebeat modules: Machine Learning jobs (#4506)

This adds support for loading ML configurations (job + datafeed) from the filebeat modules.
An example ML configuration is added to the Nginx Filebeat module. This sample applies
ML anomaly detection on the response codes.

The loading is implemented as part of the `setup` command and part of the `--setup` flag.

If a job configuration with the same ID exists, it is not overwritten, because deleting jobs
could potentially delete user data. The user should manually delete the jobs in the UI if they
want to upgrade.
(cherry picked from commit 1213483)

* use 5.5 snapshots for now

* Adjusted test for 5.x

* Another test adjustments

* On a 400 response on _xpack, assume that Xpack is not installed

* Another test fix

@tsg tsg referenced this pull request Jul 17, 2017

Closed

Filebeat modules: machine learning jobs #4680

4 of 4 tasks complete

@dedemorton dedemorton referenced this pull request Jul 25, 2017

Closed

Beats doc updates in 6.0 #4540

42 of 42 tasks complete

ramon-garcia added a commit to ramon-garcia/beats that referenced this pull request Dec 5, 2017

Filebeat modules: Machine Learning jobs (elastic#4506)
This adds support for loading ML configurations (job + datafeed) from the filebeat modules.
An example ML configuration is added to the Nginx Filebeat module. This sample applies
ML anomaly detection on the response codes.

The loading is implemented as part of the `setup` command and part of the `--setup` flag.

If a job configuration with the same ID exists, it is not overwritten, because deleting jobs
could potentially delete user data. The user should manually delete the jobs in the UI if they
want to upgrade.

@dedemorton dedemorton referenced this pull request Dec 14, 2017

Closed

Beats doc updates in 6.x #5632

37 of 37 tasks complete

@dedemorton dedemorton removed the needs_docs label Jan 22, 2018

athom added a commit to athom/beats that referenced this pull request Jan 25, 2018

Filebeat modules: Machine Learning jobs (elastic#4506)
This adds support for loading ML configurations (job + datafeed) from the filebeat modules.
An example ML configuration is added to the Nginx Filebeat module. This sample applies
ML anomaly detection on the response codes.

The loading is implemented as part of the `setup` command and part of the `--setup` flag.

If a job configuration with the same ID exists, it is not overwritten, because deleting jobs
could potentially delete user data. The user should manually delete the jobs in the UI if they
want to upgrade.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.