Possible Logstash 2.0 compatibility issue #247
Comments
|
Ahhhh odd! Much like filebeat now does courier uses JSON in the transport so you can have array fields and tags etc. At the input plugin courier uses JrJackson just like Logstash does in order to decode the network data. Could it be related to filters? Is it feasible to replay log files until you find one that reproduces and try to isolate the offending log lines maybe? Regarding the update to 2.0 compatibility there's actually no behavioural change except shutdown so I think unrelated to courier plugin, but we can keep an open mind though! |
|
Sorry I was meant to say, because JrJackson is used to deserialise and then by ES output to serialise I would expect it to be fine (unless a JrJackson bug) - this my thoughts on filters that may change the event. |
|
Thanks for the ideas @driskell ! I'll try a few simple thing like disabling all filters, then I'll narrow down from there. |
|
No luck! I disabled all the filters. I left only logstash-forwarder and courier inputs and the elasticsearch output. Same errors. |
|
Does it occur immediately? Or after a while? Have you disabled forwarder input to test that? What if you run Logstash with debug flag, it'll tell you events as it processes them so maybe point towards the problem event. |
|
It happens immediately. I'm going to test disabling the forwarder input and enable the debug flag. |
|
I know these are not beautifully formatted and little hard to read :( I enabled the debug flag, and disabled everything except for courier input, ES output, and then started a single log-courier agent. After the first message, it failed with the same error. |
|
Hi, Unfortunately, i see the same issue. Log-courier 1.8.2 and Logstash 2.0 (with the latest 1.8.2 log-courier plugin) and ES 1.5. Let me know if i can help debug this issue. |
|
Looks like some library was updated in both logstash-1.5.5 and logstash-2.0, which is not compatible. I rolled back to logstash-1.5.4 and it works fine, even with elasticsearch-2.0. |
|
Hmmm. Could someone try the file input with a file that reproduces, to see if it is indeed the fault of courier? I'll look some more into it and try some more exoteric log files and field configurations. |
|
Here is my test: cat /tmp/auth.logcat /home/logstash/test/10-input.confcat /tmp/log-courier.confcat /etc/log-courier.d/99-global.confAnd the result is: tail -2 /tmp/test.logIf i kill log-courier and change the input {} to be: And start logstash again with the same command, i don't get anything in /tmp/test.log |
|
Thanks! I just managed to reproduce it using |
|
Tracking this a little in logstash-plugins/logstash-output-elasticsearch#294 Traced it to a JrJackson issue I believe. Gonna raise there and see where we get in next day or so. If not much progress I have a workaround, it just slows the JSON processing slightly (not sure how much). It's to do with how the Courier input lets JrJackson produce Java objects rather than Ruby objects, and it's always worked fine, but it seems the updated JrJackson in Logstash 2.0.0 is having issues when serialising again to send to Elasticsearch. Just means we have to disable this and use Ruby objects. JrJackson is native Java though so it's a slight overhead, hopefully negligible if we have to disable it. |
|
I'm looking at releasing an update to Courier plugin soon to workaround this issue, and will revisit once it's fixed in JrJackson and Logstash is updated with the fixed version (which could be some time, thus the workaround plan.) |
|
👍 Also running into this. @driskell Which issue do we want pushed on thru ES support? logstash-plugins/logstash-output-elasticsearch/issues/294 |
|
Awesome, glad I ran into this thread. Been battling with this for a bit over a week. In my case it was the courier input, to an output plugin trying to use the json or json_lines codec. https://discuss.elastic.co/t/json-codec-crashing-logstash/33680 Is there any workaround (like applying some filter to achieve same result as only using Ruby objects) that we could apply in the meantime? thanks again for this project @driskell |
|
for now i just downgraded to logstash-1.5.4 (note: not logstash-1.5.5, still has same issue). |
|
I just pushed logstash-input-courier 1.8.3 Thanks |
|
Unfortunately 1.8.3 regressed the compatibility too 👎 |
|
Awesome stuff @driskell ! Thanks a ton! I'll check it out this week. |
|
After updating from 1.8.2 to 1.8.4 with logstash-2.0.0, logstash stop working at all: Uninstalling plugin doesn't help. But all start to work after completely uninstalling logstash, removing files from /opt/logstash/vendor/ and reinstalling lostash + log-courier. Seems that above described problem is solved. Another problem - logstash can't stop now: From logstash log: after 10mins log have many INFLIGHT_EVENTS_REPORT, logstash continues processing new logs. I can stop it only with sigkill. |
|
EDITED Simple low volume syslog messages with grok are fine, but when it's a 10meg file of weblogs (including unicode chars), I'm seeing corrupted objects stored in ES. Like objects aren't getting to the filter as individual json objects, but clumps with Messages are not grok'd correctly because they are malformed json. I originally thought the groks were happening but I was seeing @timestamps, paths, and other fields added by the input. Client:
Server:
Client shows no errors Server, errors like |
|
@packplusplus This was reported before in #200 but I didn't locate the problem initially. I've found the output plugin incorrectly using character length instead of byte length, so if the event going out had multibyte characters it would corrupt the protocol stream. Will push out 1.9 plugins soon. @sisoftrg Apologies for that issue - I've been separating the plugin code from Log Courier into other repositories and it looks like one of the compatibility patches didn't make it through. Specifically, the one to update the shutdown code (which is triggered differently in Logstash 2.0.0) - this will also be fixed in 1.9 plugins which I'll get out shortly. |
Fixes issue reported in driskell/log-courier#200 and driskell/log-courier#247
Increase log-courier gem dependency for byte/length fix Fixes driskell/log-courier#247 Fixes driskell/log-courier#200 1.9.0
|
@sisoftrg Regarding the To fix without reinstalling Logstash:
|
|
@driskell, Thanks. |
|
I have tried the solution on logstash 1.5.5, but I have still error: |
|
@disaster37 Can you confirm you updated the plugin to 1.9.0? As well as followed the steps for |
|
I have added on my Gemfile: And I have edit my Gemfile.jruby-1.9.lock to modify After that I have run: With that the problem persist. |
I recently upgraded a small dev cluster to logstash 2.0 and elasticsearch 2.0. We've been using logstash-forwarder in our infrastructure, but I wanted to give log-courier a short (specifically because of client-side multiline codec support!). I see that very recently a patch was dropped to add compatibility for logstash 2.0 (v1.8.2). Nevertheless, I seem to be having some issue with my configuration.
I have two logstash inputs configured.
1 - logstash forwarder
2 - log courier
My courier clients are configured the following way:
{ "general": { "log stdout": true }, "network": { "servers": [ "cp3logstash01.blah.com:18789" ], "ssl ca": "/etc/logstash-ssl-ca.crt", "timeout": 15 }, "files": [ { "fields": { "app": "blah", "env": "prod", "type": "python-verbose" }, "paths": [ "/etc/sv/blah-web/log/main/current" ] }, { "fields": { "app": "blah", "env": "prod", "type": "python-celery" }, "paths": [ "/etc/sv/blah-worker/log/main/current" ] } ] }When I start one of the log-courier process I immediately start seeing the following errors in logstash:
I'm not sure if this is being caused by the combination of logstash 2.0 and elasticsearch 2.0.
According to the release notes on logstash 2.0, the JrJackson library was upgraded.
I'm going to keep on looking into it as I have time.
Found a possibly similar issue...
elastic/logstash#3289
Thanks for the awesome project @driskell
The text was updated successfully, but these errors were encountered: