Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store message from MongoDB json logs in message field #26338

Merged
merged 4 commits into from
Jun 16, 2021
Merged

Store message from MongoDB json logs in message field #26338

merged 4 commits into from
Jun 16, 2021

Conversation

jsoriano
Copy link
Member

@jsoriano jsoriano commented Jun 16, 2021 •
edited
Loading

What does this PR do?

Follow up of #24774, store message from MongoDB json logs in the message field, and store the original log message in event.original.

Remove unused field definitions for messages.

Why is it important?

To be coherent with the information collected from plaintext logs.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc. Not needed, feature not released yet.

Related issues

@jsoriano jsoriano added review Team:Integrations Label for the Integrations team backport-v7.14.0 Automated backport with mergify labels Jun 16, 2021
@jsoriano jsoriano self-assigned this Jun 16, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 16, 2021
@jsoriano jsoriano mentioned this pull request Jun 16, 2021
Merged
6 tasks
@jsoriano jsoriano added the needs_integration_sync Changes in this PR need synced to elastic/integrations. label Jun 16, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 16, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #26338 updated

  • Start Time: 2021-06-16T11:41:13.801+0000

  • Duration: 111 min 45 sec

  • Commit: 9894f3b

Test stats 🧪

Test Results
Failed 0
Passed 14053
Skipped 2306
Total 16359

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 14053
Skipped 2306
Total 16359

tetianakravchenko
tetianakravchenko approved these changes Jun 16, 2021
View reviewed changes
Copy link
Contributor

@tetianakravchenko tetianakravchenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@tetianakravchenko
Copy link
Contributor

tetianakravchenko commented Jun 16, 2021 •
edited
Loading

@jsoriano FYI: seems that for elasticsearch slow logs there is also used message field for unparsed original log:
https://github.com/elastic/beats/blob/master/filebeat/module/elasticsearch/slowlog/test/es74_index_indexing_slowlog-json.log-expected.json#L27, not sure if there exists any conventions on field name for the original log message

@jsoriano
Copy link
Member Author

jsoriano commented Jun 16, 2021 •
edited
Loading

@jsoriano FYI: seems that for elasticsearch slow logs there is also used message field for unparsed original log:
https://github.com/elastic/beats/blob/master/filebeat/module/elasticsearch/slowlog/test/es74_index_indexing_slowlog-json.log-expected.json#L27, not sure if there exists any conventions on field name for the original log message

Umm, in this case maybe the original log line is there as message because there is no better thing to put in there, these log lines don't contain any actual message.

The convention is to use message for the relevant log message, that usually is the message without timestamp and other metadata. And to use log.original (from ECS) for the original message. Update: event.original should be used for the original message (log.original is going to be deprecated).
The original log is many times omitted, specially when all the information has been parsed and copied to other fields. Keeping the original log apart of the parsed data increases the disk size of indexes.

@jsoriano
Copy link
Member Author

Btw, message is also defined along the base fields of ECS https://www.elastic.co/guide/en/ecs/1.10/ecs-base.html.

P1llus
P1llus approved these changes Jun 16, 2021
View reviewed changes
Copy link
Member

@P1llus P1llus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one quick comment

filebeat/module/mongodb/log/ingest/pipeline-json.yml Outdated Show resolved Hide resolved
@jsoriano jsoriano merged commit b18d1f2 into elastic:master Jun 16, 2021
@jsoriano jsoriano deleted the mongodb-logs-json-message branch June 16, 2021 13:36
mergify bot pushed a commit that referenced this pull request Jun 16, 2021
@jsoriano
Store message from MongoDB json logs in message field (#26338)
ffa28b3 
(cherry picked from commit b18d1f2)
@mergify mergify bot mentioned this pull request Jun 16, 2021
Merged
jsoriano added a commit that referenced this pull request Jun 17, 2021
@mergify @jsoriano
Store message from MongoDB json logs in message field (#26338) (#26344)
48f1032 
(cherry picked from commit b18d1f2)

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
@jsoriano jsoriano mentioned this pull request Jun 17, 2021
Merged
4 tasks
@jsoriano
Copy link
Member Author

Change applied to integrations in elastic/integrations#1138.

@jsoriano jsoriano removed the needs_integration_sync Changes in this PR need synced to elastic/integrations. label Jun 17, 2021
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jun 21, 2021
@mdelapenya
Merge branch 'master' into archive-system-tests
ead2f67 
* master: (25 commits)
  Fix UBI source URL (elastic#26384)
  Skip test_rotating_file in osx and windows (elastic#26379)
  Remove outdated k8s manifests for managed elastic-agent (elastic#26368)
  Enable agent to send custom headers to kibana/ES (elastic#26275)
  [Automation] Update elastic stack version to 8.0.0-943ef2c0 for testing (elastic#26354)
  Make the Syslog input GA (elastic#26293)
  Move Kerberos FAST config flag to shared kerberos config (elastic#26141)
  Add k8s cluster identifiers (elastic#26056)
  Store message from MongoDB json logs in message field (elastic#26338)
  update threatintel ECS version (elastic#26274)
  update envoyproxy ECS version (elastic#26277)
  [Filebeat] [MongoDB] Support MongoDB 4.4 json logs (elastic#24774)
  Update go-structform to 0.0.9 (elastic#26251)
  Forward port 7.13.2 changelog to master (elastic#26323)
  Updated filter expression for filtering 86 artifacts (elastic#26313)
  Osquerybeat: Align with the rest of the beats, set the ECS version (elastic#26324)
  [Packetbeat] Add `url.extension` to Packetbeat HTTP events (elastic#25999)
  Change link to snapshots in README (elastic#26317)
  Don't include full ES index template in errors (elastic#25743)
  First refactor of the system module - system/cpu and system/core (elastic#25771)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@P1llus P1llus P1llus approved these changes

@tetianakravchenko tetianakravchenko tetianakravchenko approved these changes

Assignees

@jsoriano jsoriano

Labels
backport-v7.14.0 Automated backport with mergify review Team:Integrations Label for the Integrations team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@jsoriano @elasticmachine @tetianakravchenko @P1llus