Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
ElasticsearchWriter gives "Unexpected response code 400" with Elasticsearch 6.x #5795
Using Elasticsearch 6.0.0 gives me error code 400 on the elasticsearch feature.
Enabling debuglog shows:
Reading the docs says:
Does this feature work on Elasticsearch 6.0.0 ?
BTW, first try was with nginx in front of the elastic, but after I get the HTTP error 400, I stop the nginx and put the elastic on the lan.
thanks for reply.
The Bulk API says:
The Content-Type header should be set to application/x-ndjson
Can this be the problem?
referenced this issue
Nov 27, 2017
5.6.x works for me, I did the final release tests for 2.8 inside the Vagrant box.
Content-Type: application/json should still be supported, according to this issue it is unclear if the new content-type is really the new way to go.
rsyslog reverted the content-type change in rsyslog/rsyslog#1743
I don't think that this is related to this issue then. Still, proper tests with v6 are required.
It might be related to wrongly parsing the response body. @gunnarbeutner told me about this when looking into the InfluxDBWriter code.
Still, I'd like to see the Elasticsearch log when the status code 400 is returned.
@dnsmichi Setting up a dev ES 6 and enabled debug, the log shows:
Hope that helps.
It is kind of a ride to bring the components together for Elastic Stack 6.x inside Puppet 4, but now I see it too with a local version of Icinga/icinga-vagrant#56.
I don't think that there is a workaround, as you need to add a final "\n" to the sent body. I am working on this, still I have other projects atm. Testing a local fix.
Kibana likely adds the missing newline on its own. It might be the case that Kibana developers added this in 5.x already as they probably knew about this breaking change.
Two things for this issue:
new line terminator
While researching on the matter, this newline character terminator was expected to be there in 5.x too. For some reason, the API error handling in 6.x improved and now marks this as visible error.
Could be related: elastic/elasticsearch@c898e8a#diff-09b5159da8eeeb38a2773ab951cfa4ef
This also mentions this:
Which makes me believe that "application/json" is the current safe way to go.
also enforces a more strict content-type header. This is turned off in 5.x, but enabled in 6.x.
Further reading: https://www.elastic.co/blog/strict-content-type-checking-for-elasticsearch-rest-requests - cannot be disabled in 6.x, no possible workaround.
So I'd say we were using a bug in Elasticsearch 5.x which now has been fixed. There's no other way around than testing the snapshot packages once the coming PR is merged, and then wait for 2.8.1.
added a commit
Dec 7, 2017
referenced this issue
Dec 7, 2017
@dnsmichi Thanks for the fix.
I tested the new packege (version: v2.8.0-164-g9b75548) in my test evirenmont.
A few seconds after starting or reloading the Icinga2 service the ElasticsearchWriter discontinue writing. Here my Icinga2 ElasticsearchWriter log:
After starting icinga2, I use
nothing is showing (only after a stop of Icinga2)
debug.log shows something after a while (like a cache or something)
Using the gdb (found in Icinga2 docs)
the ElasticsearchWriter connects to elastic without any problem and without the bulk-api-error
It's a fresh installation on an updated CentOS6 VM.
Elasticsearch version 6.1.1.
The following entries are found in my debug log:
@dnsmichi Sorry, I just noticed that only some sort of data arrived in elastic (version 6.0).
Looking deeper shows error in elastic log:
So, I think my last post is incorrect about the difference between normal start and gdb start.
BTW, after deleting all indices on elastic and "normal" starting icinga2, nothing arrived in elastic,
After starting icinga2 with gdb, "message(s)" arrived in elastic and log shows:
@dnsmichi yes i have, but i have there the same errors with icingabeat and icingabeat is working fine. Hopefully it helps: