Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Adds ML jobs for access logs to Apache package #910

Merged
merged 3 commits into from Apr 19, 2021

Conversation

peteharverson
Copy link
Contributor

@peteharverson peteharverson commented Apr 14, 2021

What does this PR do?

Adds an ML module containing anomaly detection jobs for finding unusual activity in HTTP access logs to the Apache integration. Requires Kibana 7.13.0 or later.

These are the same five jobs that have previously been stored inside the ML Kibana plugin:

  • Detect unusual visitor rates
  • Detect unusual status code rates
  • Detect unusual source IPs - high distinct count of URLs
  • Detect unusual source IPs - high request rate
  • Detect low request rates

Some minor edits have been made to the previous job configurations stored in the ML Kibana plugin:

  • ID of the module is apache_data_stream compared to apache_ecs for the legacy module
  • Module and datafeed queries use data_stream.dataset: apache.access compared to event.dataset: apache.access for the legacy module
  • The ML module no longer adds its own Kibana dashboard, but instead links to the Apache logs overview dashboard which is already included in the Apache package.
  • The suffix (ECS) has been removed from the module and job description
  • _apache is appended to the IDs of the jobs in the module
  • The created_by property used for telemetry is set to ml-module-apache-access-data-stream compared to ml-module-apache-access for the legacy module

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.

How to test this PR locally

To test this PR:

  • Kibana 7.13 or later is required
  • the Apache package will first need to be added / updated from the Fleet Integrations page, or install the assets from the Settings tab for the Apache package in Fleet.
  • Have an apache data stream configured in Fleet, so that there is data in an index, such as logs-*, matching the query in the ML module JSON file:
    "query": {
      "bool": {
        "filter": [
          {
            "term": {
              "data_stream.dataset": "apache.access"
            }
          },
          {
            "exists": {
              "field": "source.address"
            }
          },
          {
            "exists": {
              "field": "url.original"
            }
          },
          {
            "exists": {
              "field": "http.response.status_code"
            }
          }
        ]
      }
    },
  • Go to the ML plugin in Kibana, and create a job, selecting the appropriate index (such as logs-*) and select the card for this new Apache access logs module:
    image
  • Create and run the jobs from the ML job wizard
  • Test that the custom URLs to the Apache logs overview dashboard and the raw data in Discover work

Related issues

elastic/package-spec#148

Screenshots

ML module is now listed in the Kibana assets section for the Apache package:
image

List of Apache jobs in the ML Job list:
image

Screenshot showing results of Apache ML jobs in the ML Anomaly Explorer:
image

@elasticmachine
Copy link

elasticmachine commented Apr 14, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #910 updated

  • Start Time: 2021-04-19T08:31:40.255+0000

  • Duration: 13 min 52 sec

  • Commit: dd59a6c

Test stats 🧪

Test Results
Failed 0
Passed 43
Skipped 0
Total 43

Trends 🧪

Image of Build Times

Image of Tests

Copy link
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rebase it against master as I pushed fix for the missing spec (ML modules).

@peteharverson peteharverson merged commit ca7f9b5 into elastic:master Apr 19, 2021
@peteharverson peteharverson deleted the ml-add-apache-module branch April 19, 2021 09:14
james-elastic pushed a commit to james-elastic/integrations that referenced this pull request Jun 30, 2021
* [ML] Adds ML jobs for access logs to Apache package

* Updates changelog to add entry for addition of ML jobs

* Formatting fix for apache-Logs-ml.json
eyalkraft pushed a commit to build-security/integrations that referenced this pull request Mar 30, 2022
* [ML] Adds ML jobs for access logs to Apache package

* Updates changelog to add entry for addition of ML jobs

* Formatting fix for apache-Logs-ml.json
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Integration:Apache
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants