Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it possible to disable host metadata #17919

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Apr 22, 2020

What does this PR do?

This provides a way to disable the addition of host metadata without introducing a breaking change. By setting @metadata.internal.distable_host_metadata=true in an event this will disable the addition of host.name by libbeat and disable add_host_metadata if it exists in the pipeline.

This can be removed in 8.0 if/when we disable host.name is libbeat and modify the default Beat configurations.

Why is it important?

ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Some data sources only forward data and it would be incorrect to set host.* fields.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Relates #13920

This provides a way to disable the addition of host metadata without introducing a breaking change.
By setting @metadata.internal.distable_host_metadata=true in an event this will disable the
addition of host.name by libbeat and disable add_host_metadata if it exists in the pipeline.

This can be removed in 8.0 if we disable host.name is libbeat and modify how the default configurations.
The PANW module always forwards data from another device so Filebeat
should not set the host fields.
@andrewkroh andrewkroh force-pushed the feature/add-host-metadata-overwrite branch from 3d01296 to 04706b0 Compare April 26, 2020 23:44
@andrewkroh andrewkroh changed the title Don't overwrite existing host metadata with add_host_metadata Make it possible to disable host metadata Apr 26, 2020
@andresrc andresrc added [zube]: Inbox Team:Integrations Label for the Integrations team labels Apr 27, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@urso
Copy link

urso commented Apr 27, 2020

One can 'disable' adding 'Host' altogether when constructing a Beat. Where would you use this changes? I'd prefer if we can do without this feature.

Instead of adding some magic fields to add_fields, we can enhance drop_fields to add a field name to the ignore list. This would allow us to ignore any field in a generic manner.

@elasticmachine
Copy link
Collaborator

💔 Build Failed

Pipeline View Test View Changes Artifacts preview stats

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Start Time: 2020-04-30T20:36:08.028+0000

  • Duration: 127 min 38 sec (7597541)

  • Commit: 04706b0

Test stats 🧪

Test Results
Failed 1
Passed 4445
Skipped 733
Total 5179

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat x-pack / runTest – nose.failure.Failure

    • Status: FAILED
    • Age: 1
    • Duration: 0
    • Error Details: Cannot subscript an existing Union. Use Union[u, t] instead.

Steps errors

Expand to view the steps failures

  • Name: Mage update build test

    • Description: mage update build test

    • Result: FAILURE

    • Duration: 6 min 50 sec<

    • Start Time: 2020-04-30T22:00:26.315+0000

  • Name: Make -C generator/_templates/metricbeat test

    • Description: make -C generator/_templates/metricbeat test

    • Result: FAILURE

    • Duration: 1 min 57 sec<

    • Start Time: 2020-04-30T22:01:17.483+0000

  • Name: Make -C generator/_templates/beat test

    • Description: make -C generator/_templates/beat test

    • Result: FAILURE

    • Duration: 1 min 55 sec<

    • Start Time: 2020-04-30T22:18:07.754+0000

Log output

Expand to view the last 100 lines of log output

[2020-04-30T22:42:09.193Z] Sending interrupt signal to process
[2020-04-30T22:42:10.532Z] failed to run compiled magefile: signal: terminated
[2020-04-30T22:42:10.659Z] script returned exit code 255
[2020-04-30T22:42:11.625Z] Recording test results
[2020-04-30T22:42:11.723Z] Stage "Heartbeat Windows" skipped due to earlier failure(s)
[2020-04-30T22:42:11.739Z] Stage "Auditbeat Windows" skipped due to earlier failure(s)
[2020-04-30T22:42:11.758Z] Stage "Functionbeat Windows" skipped due to earlier failure(s)
[2020-04-30T22:42:11.763Z] Stage "Generators Beat Mac OS X" skipped due to earlier failure(s)
[2020-04-30T22:42:12.930Z] Recording test results
[2020-04-30T22:42:14.124Z] Failed in branch Heartbeat
[2020-04-30T22:42:14.136Z] Failed in branch Auditbeat oss
[2020-04-30T22:42:14.141Z] Failed in branch Functionbeat
[2020-04-30T22:42:14.144Z] Failed in branch Generators
[2020-04-30T22:42:14.595Z] Failed in branch Elastic Agent Mac OS X
[2020-04-30T22:42:14.603Z] Failed in branch Metricbeat Mac OS X
[2020-04-30T22:42:14.704Z] Failed in branch Filebeat Mac OS X
[2020-04-30T22:42:14.704Z] Stage "Libbeat stress-tests" skipped due to earlier failure(s)
[2020-04-30T22:42:18.840Z] Failed in branch Libbeat
[2020-04-30T22:42:19.017Z] Failed in branch Winlogbeat
[2020-04-30T22:42:19.502Z] Stashed 37 file(s)
[2020-04-30T22:42:19.522Z] Archiving artifacts
[2020-04-30T22:42:20.087Z] Stashed 22 file(s)
[2020-04-30T22:42:20.363Z] Archiving artifacts
[2020-04-30T22:42:20.931Z] + curl -sSLo codecov https://codecov.io/bash
[2020-04-30T22:42:21.231Z] + FILE=auditbeat/build/coverage/full.cov
[2020-04-30T22:42:21.231Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-04-30T22:42:21.231Z] + FILE=filebeat/build/coverage/full.cov
[2020-04-30T22:42:21.231Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-04-30T22:42:21.231Z] + FILE=heartbeat/build/coverage/full.cov
[2020-04-30T22:42:21.231Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-04-30T22:42:21.231Z] + FILE=libbeat/build/coverage/full.cov
[2020-04-30T22:42:21.231Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-04-30T22:42:21.231Z] + FILE=metricbeat/build/coverage/full.cov
[2020-04-30T22:42:21.231Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-04-30T22:42:21.231Z] + FILE=packetbeat/build/coverage/full.cov
[2020-04-30T22:42:21.231Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-04-30T22:42:21.232Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-04-30T22:42:21.232Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-04-30T22:42:21.232Z] + FILE=journalbeat/build/coverage/full.cov
[2020-04-30T22:42:21.232Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-04-30T22:42:22.004Z] + curl -sSLo codecov https://codecov.io/bash
[2020-04-30T22:42:23.014Z] + FILE=auditbeat/build/coverage/full.cov
[2020-04-30T22:42:23.014Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-04-30T22:42:23.014Z] + FILE=filebeat/build/coverage/full.cov
[2020-04-30T22:42:23.014Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-04-30T22:42:23.014Z] + FILE=heartbeat/build/coverage/full.cov
[2020-04-30T22:42:23.014Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-04-30T22:42:23.014Z] + FILE=libbeat/build/coverage/full.cov
[2020-04-30T22:42:23.014Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-04-30T22:42:23.014Z] + FILE=metricbeat/build/coverage/full.cov
[2020-04-30T22:42:23.014Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-04-30T22:42:23.014Z] + FILE=packetbeat/build/coverage/full.cov
[2020-04-30T22:42:23.014Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-04-30T22:42:23.014Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-04-30T22:42:23.014Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-04-30T22:42:23.014Z] + FILE=journalbeat/build/coverage/full.cov
[2020-04-30T22:42:23.014Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-04-30T22:42:23.744Z] Failed in branch Metricbeat OSS Integration tests
[2020-04-30T22:42:25.241Z] Failed in branch Metricbeat x-pack
[2020-04-30T22:42:26.134Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats
[2020-04-30T22:42:26.620Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-04-30T22:42:26.660Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Lint
[2020-04-30T22:42:27.065Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Winlogbeat-oss
[2020-04-30T22:42:27.456Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-04-30T22:42:27.757Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-04-30T22:42:27.989Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Dockerlogbeat
[2020-04-30T22:42:28.269Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Journalbeat-oss
[2020-04-30T22:42:28.616Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-04-30T22:42:28.986Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-04-30T22:42:29.301Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Filebeat-x-pack
[2020-04-30T22:42:29.675Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-04-30T22:42:30.067Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-04-30T22:42:30.545Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-04-30T22:42:30.846Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Heartbeat-oss
[2020-04-30T22:42:31.188Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-04-30T22:42:31.738Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Filebeat-Windows
[2020-04-30T22:42:32.063Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Libbeat-x-pack
[2020-04-30T22:42:32.461Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Auditbeat-Linux
[2020-04-30T22:42:32.765Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Packetbeat-oss
[2020-04-30T22:42:33.073Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Metricbeat-Windows
[2020-04-30T22:42:33.282Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-04-30T22:42:33.694Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Filebeat-oss
[2020-04-30T22:42:34.006Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Libbeat-oss
[2020-04-30T22:42:34.258Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-04-30T22:42:34.504Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-04-30T22:42:34.669Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-04-30T22:42:34.995Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-04-30T22:42:35.554Z] + cat
[2020-04-30T22:42:35.554Z] + /usr/local/bin/runbld ./runbld-script
[2020-04-30T22:42:35.554Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-04-30T22:42:40.781Z] Body did not finish within grace period; terminating with extreme prejudice
[2020-04-30T22:42:40.994Z] ERROR: runbld post build action failed.
[2020-04-30T22:42:40.995Z] Timeout has been exceeded
[2020-04-30T22:42:42.679Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17919
[2020-04-30T22:42:43.462Z] [INFO] getVaultSecret: Getting secrets
[2020-04-30T22:42:43.707Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-04-30T22:42:45.895Z] + chmod 755 generate-build-data.sh
[2020-04-30T22:42:45.895Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-17919/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-17919/runs/6 ABORTED 7597541
[2020-04-30T22:42:46.446Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-17919/runs/6/steps/?limit=10000 -o steps-info.json
[2020-04-30T22:42:48.930Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-17919/runs/6/tests/?status=FAILED -o tests-errors.json

@andrewkroh
Copy link
Member Author

I'm going to close this change. I made a smaller change to add a Filebeat input setting to disable host.name. Please see #18159.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
libbeat Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants