Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't include full ES index template in errors #25743

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented May 17, 2021

What does this PR do?

Index templates in some beats can be very large (~1MB) and including the data in errors can use a lot of memory and also makes for very large log lines. If the error is recurring then this makes the effects worse. So this change removes the index template body from the error. Users that need to see the index template for debugging can use <beatname> export template --es.version=1.2.3.

Fixes #25540

Why is it important?

~1MB log lines are difficult to work with.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 17, 2021
@andrewkroh andrewkroh added bug libbeat Team:Elastic-Agent Label for the Agent team labels May 17, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 17, 2021
@andrewkroh andrewkroh force-pushed the bugfix/libbeat/omit-index-template-from-err branch from 7ac6e07 to 54cbd0a Compare May 17, 2021 19:34
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@elasticmachine
Copy link
Collaborator

elasticmachine commented May 17, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: andrewkroh commented: run tests

  • Start Time: 2021-06-14T18:07:52.435+0000

  • Duration: 64 min 18 sec

  • Commit: b1865f9

Test stats 🧪

Test Results
Failed 0
Passed 243
Skipped 24
Total 267

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 243
Skipped 24
Total 267

@mergify
Copy link
Contributor

mergify bot commented May 19, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b bugfix/libbeat/omit-index-template-from-err upstream/bugfix/libbeat/omit-index-template-from-err
git merge upstream/master
git push upstream bugfix/libbeat/omit-index-template-from-err

@andrewkroh andrewkroh requested a review from a team June 9, 2021 23:53
Copy link
Contributor

@ph ph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, nice addition

Index templates in some beats can be very large (~1MB) and including the data in errors can use a lot of memory and also makes for very large log lines. If the error is recurring then this makes the effects worse. So this change removes the index template body from the error. Users that need to see the index template for debugging can use `<beatname> export template --es.version=1.2.3`.

Fixes elastic#25540
@andrewkroh andrewkroh force-pushed the bugfix/libbeat/omit-index-template-from-err branch from 5e2d42e to 017d150 Compare June 14, 2021 12:26
@mergify
Copy link
Contributor

mergify bot commented Jun 14, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b bugfix/libbeat/omit-index-template-from-err upstream/bugfix/libbeat/omit-index-template-from-err
git merge upstream/master
git push upstream bugfix/libbeat/omit-index-template-from-err

@andrewkroh andrewkroh added the backport-v7.14.0 Automated backport with mergify label Jun 14, 2021
@andrewkroh
Copy link
Member Author

andrewkroh commented Jun 14, 2021

The build was green, but uploading failed.

[2021-06-14T15:42:14.912Z] + gsutil -m -q cp -a public-read XX gs://beats-ci-temp/ci/cache/
[2021-06-14T15:42:16.354Z] ServiceException: 401 Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.
[2021-06-14T15:42:16.354Z] CommandException: 1 file/object could not be transferred.
script returned exit code 1

@andrewkroh
Copy link
Member Author

run tests

@andrewkroh andrewkroh merged commit 766e303 into elastic:master Jun 15, 2021
mergify bot pushed a commit that referenced this pull request Jun 15, 2021
* Don't include full ES index template in errors

Index templates in some beats can be very large (~1MB) and including the data in errors can use a lot of memory and also makes for very large log lines. If the error is recurring then this makes the effects worse. So this change removes the index template body from the error. Users that need to see the index template for debugging can use `<beatname> export template --es.version=1.2.3`.

Fixes #25540

* Update expected log messages in tests

(cherry picked from commit 766e303)

# Conflicts:
#	libbeat/template/load.go
michalpristas pushed a commit to michalpristas/beats that referenced this pull request Jun 17, 2021
* Don't include full ES index template in errors

Index templates in some beats can be very large (~1MB) and including the data in errors can use a lot of memory and also makes for very large log lines. If the error is recurring then this makes the effects worse. So this change removes the index template body from the error. Users that need to see the index template for debugging can use `<beatname> export template --es.version=1.2.3`.

Fixes elastic#25540

* Update expected log messages in tests
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jun 21, 2021
* master: (25 commits)
  Fix UBI source URL (elastic#26384)
  Skip test_rotating_file in osx and windows (elastic#26379)
  Remove outdated k8s manifests for managed elastic-agent (elastic#26368)
  Enable agent to send custom headers to kibana/ES (elastic#26275)
  [Automation] Update elastic stack version to 8.0.0-943ef2c0 for testing (elastic#26354)
  Make the Syslog input GA (elastic#26293)
  Move Kerberos FAST config flag to shared kerberos config (elastic#26141)
  Add k8s cluster identifiers (elastic#26056)
  Store message from MongoDB json logs in message field (elastic#26338)
  update threatintel ECS version (elastic#26274)
  update envoyproxy ECS version (elastic#26277)
  [Filebeat] [MongoDB] Support MongoDB 4.4 json logs (elastic#24774)
  Update go-structform to 0.0.9 (elastic#26251)
  Forward port 7.13.2 changelog to master (elastic#26323)
  Updated filter expression for filtering 86 artifacts (elastic#26313)
  Osquerybeat: Align with the rest of the beats, set the ECS version (elastic#26324)
  [Packetbeat] Add `url.extension` to Packetbeat HTTP events (elastic#25999)
  Change link to snapshots in README (elastic#26317)
  Don't include full ES index template in errors (elastic#25743)
  First refactor of the system module - system/cpu and system/core (elastic#25771)
  ...
andrewkroh added a commit that referenced this pull request Jun 30, 2021
…#26318)

* Don't include full ES index template in errors (#25743)

Index templates in some beats can be very large (~1MB) and including the data in errors can use a lot of memory and also makes for very large log lines. If the error is recurring then this makes the effects worse. So this change removes the index template body from the error. Users that need to see the index template for debugging can use `<beatname> export template --es.version=1.2.3`.

Fixes #25540

* Update expected log messages in tests

(cherry picked from commit 766e303)

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v7.14.0 Automated backport with mergify bug libbeat review Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

libbeat 7.12.1 default logging logs excessive amount of data when failing to create index template
3 participants