Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Filebeat] Threatintel compatibility updates #27323

Merged
merged 9 commits into from
Sep 7, 2021
Merged

[Filebeat] Threatintel compatibility updates #27323

merged 9 commits into from
Sep 7, 2021

Conversation

rylnd
Copy link
Contributor

@rylnd rylnd commented Aug 11, 2021 •
edited
Loading

What does this PR do?

This is a draft of the work discussed between @P1llus and myself. While the plan is to migrate most of this functionality to integration packages, there are some incompatibilities/logical errors that can/should be cleaned up in these modules. At a high level, we've outlined the following changes:

  • removing indicator.domain, which has been deprecated in favor of indicator.url.domain
  • moving from event.reference to indicator.reference, as it's useful for investigation and event.* fields aren't copied as part of enrichment
  • moving from threatintel.indicator to threat.indicator
  • moving any non-ECS threatintel.[MODULE] fieldsets to not be nested under threatintel

Why is it important?

These changes will allow filebeat 7.15 users to ingest CTI data compatible with ECS 1.11.

Remaining work

  • verify that our geo_point fields are being ingested correctly
  • updating of this field to address https://github.com/elastic/security-team/issues/1494
    • the abuseurl module uses urlhaus_reference to populate its analogous reference field. The test data doesn't indicate so, but if that exists for abusemalware as well, that's a simple fix. If that's not present, the best solution is likely to drop that field from the module.
  • verification from a beats developer 😬

Work determined to be unnecessary for 7.15

  • updating all threatintel modules to populate threat.indicator instead of threatintel.indicator
  • moving threatintel.[MODULE] fieldsets to the root level

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@rylnd
First pass on updating filebeat threatintel logic for ECS 1.11
a0fcab1 
This only covers modules starting with an a; the rest will follow
shortly. In general, these changes address the following goals:

* preference for indicator.url.domain, and deprecation of
  indicator.domain
* moving from event.reference to indicator.reference
@rylnd
Move remaining modules from indicator.domain -> indicator.url.domain
a65bb59 
Along with conditional checks to ensure we're not overwriting the
relevant uri_parts data from earlier in the pipeline.
@rylnd
Update indicator.reference in relevant modules
9c44a14
@rylnd rylnd requested a review from P1llus August 11, 2021 19:32
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 11, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Aug 11, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-09-07T13:53:40.492+0000

  • Duration: 101 min 47 sec

  • Commit: 5d7443c

Test stats 🧪

Test Results
Failed 0
Passed 15220
Skipped 2314
Total 17534

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 15220
Skipped 2314
Total 17534

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 20, 2021
@P1llus
Copy link
Member

P1llus commented Aug 31, 2021

Adding a comment here, some of the remaining work will be in a separate PR. We wont be changing threatintel.* to threat.*, or removing the nested fields before 7.16.

Will work with @rylnd to get this merged this week

@P1llus P1llus marked this pull request as ready for review September 2, 2021 14:41
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@P1llus
Copy link
Member

P1llus commented Sep 7, 2021

/test

@P1llus P1llus removed their request for review September 7, 2021 14:16
@P1llus P1llus merged commit 4be2694 into master Sep 7, 2021
@P1llus P1llus added backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify labels Sep 7, 2021
mergify bot pushed a commit that referenced this pull request Sep 7, 2021
@rylnd
[Filebeat] Threatintel compatibility updates (#27323)
f360971 
* First pass on updating filebeat threatintel logic for ECS 1.11

This only covers modules starting with an a; the rest will follow
shortly. In general, these changes address the following goals:

* preference for indicator.url.domain, and deprecation of
  indicator.domain
* moving from event.reference to indicator.reference

* Move remaining modules from indicator.domain -> indicator.url.domain

Along with conditional checks to ensure we're not overwriting the
relevant uri_parts data from earlier in the pipeline.

* Update indicator.reference in relevant modules

* Fix missing prefix in target field

* linting and apply new testfiles

* Run `make update` in filebeat

* fixing duplicate fields

* mage fmt update

* linting

Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
(cherry picked from commit 4be2694)

# Conflicts:
#	x-pack/filebeat/module/threatintel/abuseurl/ingest/pipeline.yml
#	x-pack/filebeat/module/threatintel/anomali/ingest/pipeline.yml
@mergify mergify bot mentioned this pull request Sep 7, 2021
Merged
mergify bot pushed a commit that referenced this pull request Sep 7, 2021
@rylnd
[Filebeat] Threatintel compatibility updates (#27323)
ff62211 
* First pass on updating filebeat threatintel logic for ECS 1.11

This only covers modules starting with an a; the rest will follow
shortly. In general, these changes address the following goals:

* preference for indicator.url.domain, and deprecation of
  indicator.domain
* moving from event.reference to indicator.reference

* Move remaining modules from indicator.domain -> indicator.url.domain

Along with conditional checks to ensure we're not overwriting the
relevant uri_parts data from earlier in the pipeline.

* Update indicator.reference in relevant modules

* Fix missing prefix in target field

* linting and apply new testfiles

* Run `make update` in filebeat

* fixing duplicate fields

* mage fmt update

* linting

Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
(cherry picked from commit 4be2694)

# Conflicts:
#	x-pack/filebeat/module/threatintel/abuseurl/ingest/pipeline.yml
#	x-pack/filebeat/module/threatintel/anomali/ingest/pipeline.yml
@mergify mergify bot mentioned this pull request Sep 7, 2021
Merged
@rylnd rylnd deleted the threatintel-compatibility-updates branch September 7, 2021 18:27
P1llus added a commit that referenced this pull request Sep 7, 2021
@mergify @rylnd @P1llus
[7.x](backport #27323) [Filebeat] Threatintel compatibility updates (#…
4b6538b 
…27777)

* [Filebeat] Threatintel compatibility updates (#27323)

* First pass on updating filebeat threatintel logic for ECS 1.11

This only covers modules starting with an a; the rest will follow
shortly. In general, these changes address the following goals:

* preference for indicator.url.domain, and deprecation of
  indicator.domain
* moving from event.reference to indicator.reference

* Move remaining modules from indicator.domain -> indicator.url.domain

Along with conditional checks to ensure we're not overwriting the
relevant uri_parts data from earlier in the pipeline.

* Update indicator.reference in relevant modules

* Fix missing prefix in target field

* linting and apply new testfiles

* Run `make update` in filebeat

* fixing duplicate fields

* mage fmt update

* linting

Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
(cherry picked from commit 4be2694)

# Conflicts:
#	x-pack/filebeat/module/threatintel/abuseurl/ingest/pipeline.yml
#	x-pack/filebeat/module/threatintel/anomali/ingest/pipeline.yml

* fixing mergify conflicts

Co-authored-by: Ryland Herrick <ryalnd@gmail.com>
Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
P1llus added a commit that referenced this pull request Sep 7, 2021
@mergify @rylnd @P1llus
[7.15](backport #27323) [Filebeat] Threatintel compatibility updates (#…
524e0e5 
…27778)

* [Filebeat] Threatintel compatibility updates (#27323)

* First pass on updating filebeat threatintel logic for ECS 1.11

This only covers modules starting with an a; the rest will follow
shortly. In general, these changes address the following goals:

* preference for indicator.url.domain, and deprecation of
  indicator.domain
* moving from event.reference to indicator.reference

* Move remaining modules from indicator.domain -> indicator.url.domain

Along with conditional checks to ensure we're not overwriting the
relevant uri_parts data from earlier in the pipeline.

* Update indicator.reference in relevant modules

* Fix missing prefix in target field

* linting and apply new testfiles

* Run `make update` in filebeat

* fixing duplicate fields

* mage fmt update

* linting

Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
(cherry picked from commit 4be2694)

# Conflicts:
#	x-pack/filebeat/module/threatintel/abuseurl/ingest/pipeline.yml
#	x-pack/filebeat/module/threatintel/anomali/ingest/pipeline.yml

* fixing mergify conflicts

Co-authored-by: Ryland Herrick <ryalnd@gmail.com>
Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Sep 9, 2021
@mdelapenya
Merge branch 'master' into 27608-fix-elastic-agent-build
16ae989 
* master: (39 commits)
  [Heartbeat] Move JSON tests from python->go (elastic#27816)
  docs: simplify permissions for Dockerfile COPY (elastic#27754)
  Osquerybeat: Fix osquery logger plugin severy levels mapping (elastic#27789)
  [Filebeat] Update compatibility function to remove processor description on ES < 7.9.0 (elastic#27774)
  warn log entry and no validation failure when both queue_url and buck… (elastic#27612)
  libbeat/cmd/instance: ensure test config file has appropriate permissions (elastic#27178)
  [Heartbeat] Add httpcommon options to ZipURL (elastic#27699)
  Add a header round tripper option to httpcommon (elastic#27509)
  [Elastic Agent] Add validation to ensure certificate paths are absolute. (elastic#27779)
  Rename dashboards according to module.yml files for master (elastic#27749)
  Refactor vagrantfile, add scripts for provisioning with docker/kind (elastic#27726)
  Accept syslog dates with leading 0 (elastic#27775)
  [Filebeat] Add timezone config option to decode_cef and syslog input (elastic#27727)
  [Filebeat] Threatintel compatibility updates (elastic#27323)
  Add support for ephemeral containers in elastic agent dynamic provider (elastic#27707)
  [Filebeat] Integration tests in CI for AWS-S3 input (elastic#27491)
  Fix flakyness of TestFilestreamEmptyLine (elastic#27705)
  [Filebeat] kafka v2 using parsers (elastic#27335)
  Update Kafka version parsing / supported range (elastic#27720)
  Update Sarama to 1.29.1 (elastic#27717)
  ...
Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021
@rylnd @P1llus
[Filebeat] Threatintel compatibility updates (elastic#27323)
1806da0 
* First pass on updating filebeat threatintel logic for ECS 1.11

This only covers modules starting with an a; the rest will follow
shortly. In general, these changes address the following goals:

* preference for indicator.url.domain, and deprecation of
  indicator.domain
* moving from event.reference to indicator.reference

* Move remaining modules from indicator.domain -> indicator.url.domain

Along with conditional checks to ensure we're not overwriting the
relevant uri_parts data from earlier in the pipeline.

* Update indicator.reference in relevant modules

* Fix missing prefix in target field

* linting and apply new testfiles

* Run `make update` in filebeat

* fixing duplicate fields

* mage fmt update

* linting

Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marc-gr marc-gr marc-gr approved these changes

Assignees

@P1llus P1llus

Labels
backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify enhancement v7.15.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@rylnd @elasticmachine @P1llus @marc-gr @jsoriano