Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for multi-core thread_id in filebeat postgresql module #9482

Merged
merged 6 commits into from Dec 13, 2018

Conversation

Projects
None yet
3 participants
@kaiyan-sheng
Copy link
Contributor

commented Dec 11, 2018

Currently we only support postgresql log looks like:

2018-11-17 03:14:01 EST [835] LOG: autovacuum launcher started

But this will fail when parsing postgresql log messages from multi-core machines which looks like:

2018-11-17 03:14:01 EST [835-1] LOG: autovacuum launcher started

This PR is to add optional core id after the original thread id to parse multi-core postgresql logs. Please see more details in #9156

closes #9156

@@ -83,6 +83,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha1...master[Check the HEAD d
- Added `detect_null_bytes` selector to detect null bytes from a io.reader. {pull}9210[9210]
- Added `syslog_host` variable to HAProxy module to allow syslog listener to bind to configured host. {pull}9366[9366]
- Added support on Traefik for Common Log Format and Combined Log Format mixed which is the default Traefik format {issue}8015[8015] {issue}6111[6111] {pull}8768[8768].
- Add support for multi-core thread_id in postgresql module {issue}9156[9156]

This comment has been minimized.

Copy link
@jsoriano

jsoriano Dec 11, 2018

Member

Add also a reference to this PR please

This comment has been minimized.

Copy link
@kaiyan-sheng

kaiyan-sheng Dec 11, 2018

Author Contributor

yep! will do!

@@ -0,0 +1,19 @@
2018-11-17 13:14:01 CEST [12975-1] [unknown]@[unknown] LOG: incomplete startup packet

This comment has been minimized.

Copy link
@jsoriano

jsoriano Dec 11, 2018

Member

Add the expected file for this one please, I mentioned to remove the other one that was not related to this change.

This comment has been minimized.

Copy link
@kaiyan-sheng

kaiyan-sheng Dec 11, 2018

Author Contributor

@jsoriano yes! I added it in the previous commit but the @timestamp failed on check job. That's why I tried to remove it to see if the check job will pass :-) I will definitely put it back in.

@jsoriano
Copy link
Member

left a comment

LGTM, wait to jenkins, and please add a comment with a reference to the issue to fix the timestamps thing, thanks!

@kaiyan-sheng

This comment has been minimized.

Copy link
Contributor Author

commented Dec 11, 2018

@jsoriano Thanks for your help. I just created an issue for the timestamp problem: #9494

@kaiyan-sheng kaiyan-sheng added review and removed in progress labels Dec 11, 2018

@ruflin

ruflin approved these changes Dec 12, 2018

Copy link
Collaborator

left a comment

LGTM, but needs a rebase.

Could you also update the PR description with what was changed in the PR? Please keep the links to the Github issues ;-)

kaiyan-sheng added some commits Dec 12, 2018

@kaiyan-sheng

This comment has been minimized.

Copy link
Contributor Author

commented Dec 12, 2018

@ruflin Sounds good! I just updated the PR description. Thanks!

@kaiyan-sheng kaiyan-sheng merged commit 384b11b into elastic:master Dec 13, 2018

3 of 4 checks passed

beats-ci Build finished.
Details
CLA Commit author is a member of Elasticsearch
Details
Hound No violations found. Woof!
continuous-integration/travis-ci/pr The Travis CI build passed
Details

kaiyan-sheng added a commit to kaiyan-sheng/beats that referenced this pull request Dec 13, 2018

Add support for multi-core thread_id in filebeat postgresql module (e…
…lastic#9482)

* Add support for multi-core thread_id in filebeat postgresql module

* Change timestamp for test log file

* update changelog and add multi-core.log-expected.json

* Change timestamp in example log file

* Update fields.go

(cherry picked from commit 384b11b)

kaiyan-sheng added a commit that referenced this pull request Dec 17, 2018

Cherry-pick to 6.x: Support for multi-core thread_id in filebeat post…
…gresql module (#9482) (#9528)

* Add support for multi-core thread_id in filebeat postgresql module (#9482)

* Add support for multi-core thread_id in filebeat postgresql module

* Change timestamp for test log file

* update changelog and add multi-core.log-expected.json

* Change timestamp in example log file

* Update fields.go

(cherry picked from commit 384b11b)

* Run make update

* Run test with 6.x Elasticsearch

@kaiyan-sheng kaiyan-sheng deleted the kaiyan-sheng:multi-core-postgresql branch Jan 31, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.