Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filebeat Log Collection Halting : ERR Fail to convert the event to JSON: reflect: call of reflect.Value.IsNil on zero Value #1410

Closed
david-rooke opened this issue Apr 18, 2016 · 8 comments
Assignees
Labels

Comments

@david-rooke
Copy link

I am seeing that log collection is halting periodically when sending Datastax Enterprise Cassandra logs to logstash. Issue was originally seen in version 1.1.0 and 1.2.1 with this error:

ERR Fail to convert the event to JSON: reflect: call of reflect.Value.IsNil on zero Value

Installing a 5.0.0 nightly build didn't resolve the problem or provide any additional output. Indeed, the service tends to die silently...

Additional information and background can be found here (https://discuss.elastic.co/t/filebeat-1-1-0-log-collection-halting/47130/15)

  • Version: 1.1.0, 1.2.1, 5.0.0 (currently using 5.0.0 nightly)
  • Operating System: Ubuntu 14.04
  • Steps to Reproduce: I have attached a sample sanitised log from the server and filebeat shipper.

sample_logs.tar.gz

@tsg
Copy link
Contributor

tsg commented Apr 18, 2016

@david-rooke can you also provide your filebeat.yml? I know you posted it in discuss, but I want to make sure we have your up-to-date version before we try to reproduce this. Thanks!

@david-rooke
Copy link
Author

@tsg config attached...
filebeat_conf.tar.gz

@tsg
Copy link
Contributor

tsg commented Apr 18, 2016

@david-rooke we have a suspicion that the multiline implementation might be causing this, but in the sample logs you provided there doesn't seem to be any multiline events (lines starting with a white space).

  1. If you can reproduce this relatively easily, can you try removing the multiline configuration to confirm this theory.
  2. Can you provide a sample log that contains such a multiline event?

Thanks!

@tsg
Copy link
Contributor

tsg commented Apr 18, 2016

Ah, nevermind point 2, I see now that it contains an exception, it was just my regexp being broken. Point 1 could still be interesting.

@david-rooke
Copy link
Author

Well, I've seen this behaviour whether multiline has been enabled or not.

@tsg
Copy link
Contributor

tsg commented Apr 18, 2016

Thanks, that's useful information.

@tsg
Copy link
Contributor

tsg commented Apr 18, 2016

I managed to reproduce the issue. It's related to the fact that there are two outputs enabled (console and logstash). It can happen that they try to modify the same event concurrently which results in a panic. Stack trace here: https://gist.github.com/tsg/fbfdac8ea8a0b1f74f17e4a7c4715dee

We probably need to do deep copying of the events before passing them to multiple outputs.

@david-rooke As a workaround, I'd recommend disabling the console output for the moment.

@urso
Copy link

urso commented Apr 18, 2016

True, the logstash output adds meta-data to the event without copying it. potential race on map.

@urso urso self-assigned this Apr 18, 2016
@monicasarbu monicasarbu added P1 and removed priority labels Apr 19, 2016
@tsg tsg closed this as completed in #1428 Apr 20, 2016
urso pushed a commit to urso/beats that referenced this issue Apr 20, 2016
logstash output used to add metadata to an event. With potentially having
multiple outputs configured this is some bad practice. Instead the logstash
output plugin will add '@metadata' to the event when encoding and only if
'@metadata' is not already present in event
tsg pushed a commit that referenced this issue Apr 21, 2016
logstash output used to add metadata to an event. With potentially having
multiple outputs configured this is some bad practice. Instead the logstash
output plugin will add '@metadata' to the event when encoding and only if
'@metadata' is not already present in event
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants