Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ECS information to logger instance #18820

Closed
wants to merge 4 commits into from

Conversation

simitt
Copy link
Contributor

@simitt simitt commented May 28, 2020

What does this PR do?

The PR adds a ECSEnabled() method to *logp.Logger instances.

Why is it important?

Since introducing the logging.ecs setting some minimal required information is logged in ECS format when enabled. To continue work for ECS conformant logging without introducing conflicts with existing log formats, developers need to be able to check for ECS being enabled or not whenever they log information.

Checklist

  • My code follows the style guidelines of this project
    - [ ] I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
    - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

run go test ./libbeat/logp/...

Related issues

Derive information whether ECS logging is enabled from any
logger instance as requirement to make decisions about logging keys to
use.

closes elastic#18815
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 28, 2020
@simitt simitt added enhancement Team:Platforms Label for the Integrations - Platforms team labels May 28, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 28, 2020
@exekias exekias added Team:Integrations Label for the Integrations team and removed Team:Platforms Label for the Integrations - Platforms team labels May 28, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 28, 2020

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Start Time: 2020-06-15T22:36:52.320+0000

  • Duration: 123 min 14 sec

Test stats 🧪

Test Results
Failed 2
Passed 8002
Skipped 1518
Total 9522

Test errors

Expand to view the tests failures

  • Name: Build and Test / Metricbeat OSS Unit tests / test_process – test_system.Test

    • Age: 1
    • Duration: 1.809
    • Error Details: False is not true : fd not found in any process events
  • Name: Build and Test / Filebeat Mac OS X / test_clean_removed_with_clean_inactive – test_registrar.Test

    • Age: 1
    • Duration: 1.379
    • Error Details:
      -------------------- >> begin captured stdout << ---------------------
      registry size: 2
      registry size after remove: 2

--------------------- >> end captured stdout << ----------------------

Steps errors

Expand to view the steps failures

  • Name: Mage build unitTest

    • Description: mage build unitTest

    • Duration: 12 min 46 sec

    • Start Time: 2020-06-15T23:31:04.411+0000

    • log

  • Name: Recursively delete the current directory from the workspace

    • Description: script returned exit code 1

    • Duration: 0 min 14 sec

    • Start Time: 2020-06-15T23:44:38.670+0000

    • log

  • Name: Install Go 1.13.10

    • Description: .ci/scripts/install-go.sh

    • Duration: 1 min 8 sec

    • Start Time: 2020-06-15T23:26:05.184+0000

    • log

  • Name: Mage build unitTest

    • Description: mage build unitTest

    • Duration: 11 min 56 sec

    • Start Time: 2020-06-15T23:29:17.377+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-06-16T00:39:33.555Z] + chmod +x /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/bin/docker-compose
[2020-06-16T00:39:34.062Z] + .ci/scripts/install-terraform.sh
[2020-06-16T00:39:34.062Z] + MSG='parameter missing.'
[2020-06-16T00:39:34.062Z] + TERRAFORM_VERSION=0.12.24
[2020-06-16T00:39:34.062Z] + HOME=/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820
[2020-06-16T00:39:34.063Z] + TERRAFORM_CMD=/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/bin/terraform
[2020-06-16T00:39:34.063Z] ++ tr '[:upper:]' '[:lower:]'
[2020-06-16T00:39:34.063Z] ++ uname -s
[2020-06-16T00:39:34.063Z] + OS=linux
[2020-06-16T00:39:34.063Z] + mkdir -p /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/bin
[2020-06-16T00:39:34.063Z] + curl -sSLo - https://releases.hashicorp.com/terraform/0.12.24/terraform_0.12.24_linux_amd64.zip
[2020-06-16T00:39:34.639Z] ++ dirname /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/bin/terraform
[2020-06-16T00:39:34.639Z] + unzip -o /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/bin/terraform.zip -d /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/bin
[2020-06-16T00:39:34.639Z] Archive:  /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/bin/terraform.zip
[2020-06-16T00:39:35.216Z]   inflating: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/bin/terraform  
[2020-06-16T00:39:35.216Z] + rm /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/bin/terraform.zip
[2020-06-16T00:39:35.216Z] + chmod +x /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/bin/terraform
[2020-06-16T00:39:35.680Z] + make mage
[2020-06-16T00:39:35.680Z] Installing mage v1.9.0 from vendor dir.
[2020-06-16T00:39:36.648Z] /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/.magefile cleaned
[2020-06-16T00:39:37.110Z] + git config --get user.email
[2020-06-16T00:39:37.110Z] + [ -z  ]
[2020-06-16T00:39:37.110Z] + git config user.email beatsmachine@users.noreply.github.com
[2020-06-16T00:39:37.110Z] + git config user.name beatsmachine
[2020-06-16T00:39:37.758Z] + .ci/scripts/terraform-cleanup.sh x-pack/metricbeat
[2020-06-16T00:39:37.758Z] + DIRECTORY=x-pack/metricbeat
[2020-06-16T00:39:37.758Z] + FAILED=0
[2020-06-16T00:39:37.758Z] ++ find x-pack/metricbeat -name terraform.tfstate
[2020-06-16T00:39:37.758Z] + exit 0
[2020-06-16T00:39:38.530Z] + curl -sSLo codecov https://codecov.io/bash
[2020-06-16T00:39:38.803Z] + FILE=auditbeat/build/coverage/full.cov
[2020-06-16T00:39:38.804Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-06-16T00:39:38.804Z] + FILE=filebeat/build/coverage/full.cov
[2020-06-16T00:39:38.804Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-06-16T00:39:38.804Z] + FILE=heartbeat/build/coverage/full.cov
[2020-06-16T00:39:38.804Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-06-16T00:39:38.804Z] + FILE=libbeat/build/coverage/full.cov
[2020-06-16T00:39:38.804Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-06-16T00:39:38.804Z] + FILE=metricbeat/build/coverage/full.cov
[2020-06-16T00:39:38.804Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-06-16T00:39:38.804Z] + FILE=packetbeat/build/coverage/full.cov
[2020-06-16T00:39:38.804Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-06-16T00:39:38.804Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-06-16T00:39:38.804Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-06-16T00:39:38.804Z] + FILE=journalbeat/build/coverage/full.cov
[2020-06-16T00:39:38.804Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-06-16T00:39:40.349Z] Failed in branch Metricbeat x-pack
[2020-06-16T00:39:40.845Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats
[2020-06-16T00:39:41.316Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-06-16T00:39:41.337Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Lint
[2020-06-16T00:39:41.771Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-06-16T00:39:42.224Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-06-16T00:39:42.840Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-06-16T00:39:43.146Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Libbeat-x-pack
[2020-06-16T00:39:43.531Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-06-16T00:39:43.810Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-06-16T00:39:44.244Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-06-16T00:39:44.671Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-06-16T00:39:45.101Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-06-16T00:39:45.508Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-06-16T00:39:45.938Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-06-16T00:39:46.262Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-06-16T00:39:46.568Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-06-16T00:39:47.181Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-06-16T00:39:47.542Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-06-16T00:39:47.877Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Filebeat-Windows
[2020-06-16T00:39:48.162Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Winlogbeat-oss
[2020-06-16T00:39:48.480Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Dockerlogbeat
[2020-06-16T00:39:48.840Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Heartbeat-oss
[2020-06-16T00:39:49.433Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Journalbeat-oss
[2020-06-16T00:39:49.730Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-06-16T00:39:50.086Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-06-16T00:39:50.510Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-06-16T00:39:51.016Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-06-16T00:39:51.358Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Metricbeat-Windows
[2020-06-16T00:39:51.939Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-06-16T00:39:52.337Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-06-16T00:39:52.818Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-06-16T00:39:53.153Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-06-16T00:39:53.508Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Packetbeat-oss
[2020-06-16T00:39:54.071Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Filebeat-x-pack
[2020-06-16T00:39:54.677Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Filebeat-oss
[2020-06-16T00:39:55.136Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-06-16T00:39:55.553Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-06-16T00:39:55.948Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Libbeat-oss
[2020-06-16T00:39:56.521Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-06-16T00:39:56.982Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-06-16T00:39:57.601Z] + cat
[2020-06-16T00:39:57.602Z] + /usr/local/bin/runbld ./runbld-script
[2020-06-16T00:39:57.602Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-06-16T00:39:57.831Z] Body did not finish within grace period; terminating with extreme prejudice
[2020-06-16T00:39:57.987Z] ERROR: runbld post build action failed.
[2020-06-16T00:39:57.987Z] Timeout has been exceeded
[2020-06-16T00:40:04.835Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18820
[2020-06-16T00:40:05.057Z] [INFO] getVaultSecret: Getting secrets
[2020-06-16T00:40:05.210Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-16T00:40:06.606Z] + chmod 755 generate-build-data.sh
[2020-06-16T00:40:06.606Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18820/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18820/runs/6 ABORTED 7393997
[2020-06-16T00:40:06.606Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18820/runs/6/steps/?limit=10000 -o steps-info.json
[2020-06-16T00:40:13.090Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18820/runs/6/tests/?status=FAILED -o tests-errors.json

@simitt
Copy link
Contributor Author

simitt commented Jun 2, 2020

test failures seem unrelated

@simitt simitt requested a review from urso June 2, 2020 09:13
@simitt
Copy link
Contributor Author

simitt commented Jun 4, 2020

@urso I'd appreciate a review on this.

@urso
Copy link

urso commented Jun 4, 2020

Why do we want developers to check if ecs enabled? We should encourage structured logging always, but with this check it looks like we will encourage developers to use different 'schemas'.

With the current state of structured logging in Beats would we consider a log field being renamed to ECS a breaking change?

@simitt
Copy link
Contributor Author

simitt commented Jun 4, 2020

IMO renaming a field would be fine, but some changes could lead to type changes, e.g. an when an error was logged as { "error": "foo" } and with ECS gets logged as { "error": { "message": "foo" }}. If users ingest the logs this might break some setups.
As long as we have the ecs logging configurable and are not fully switched over this helps avoid such breaking changes.

@urso
Copy link

urso commented Jun 4, 2020

The error type is internal to logp, right? You think we will have other breaking changes right in Beats/apm-server?

@simitt
Copy link
Contributor Author

simitt commented Jun 4, 2020

Eg. if the logger is used like this logger.Errorw("log message", "error", err.Error),
the produced log line would look like { "message": "log message, "error": "foo" },
switching to ECS it looks like { "message": "log message, "error": { "message": "foo" } }.

When shipping such logs to ES one would need a new mapping for error as it changes from keyword to object. That's what I consider a breaking change.
I have a draft PR up in APM Server to ensure ECS compliant logging, where I encountered 2 breaking changes elastic/apm-server#3829

@simitt
Copy link
Contributor Author

simitt commented Jun 16, 2020

After some further discussions, closing this PR. At the moment there is no commitment for not changing the logging format or structure, therefore we will not add the ECSEnabled() method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Figure out if ECS logging is enabled when holding a logger instance
5 participants