ElasticsearchWriter gives "Unexpected response code 400" with Elasticsearch 6.x #5795
Comments
|
I should work, I see nothing in the breaking changes that should affect the elasticsearchwriter. But when we implemented this feature, there was no elasticsearch 6 yet, so it wasn't tested with that version. |
|
Dear Crunsher, thanks for reply.
The Bulk API says:
The Content-Type header should be set to application/x-ndjson Can this be the problem? |
|
The same error happens on ES 5.6, too, so this is not ES 6.0 specific. |
gunnarbeutner
added
bug
|
@BjoernRobbe Oh I overlooked this, we are using application/json When developing I was using 5.4 |
|
@Crunsher Looking at the docs for the Bulk API they started with Version 5.3 with this |
|
¯_(ツ)_/¯ I'll investigate once I'm on a better connection and can update my elastic vagrant box |
|
5.6.x works for me, I did the final release tests for 2.8 inside the Vagrant box. |
|
Content-Type: application/json should still be supported, according to this issue it is unclear if the new content-type is really the new way to go. rsyslog reverted the content-type change in rsyslog/rsyslog#1743 I don't think that this is related to this issue then. Still, proper tests with v6 are required. It might be related to wrongly parsing the response body. @gunnarbeutner told me about this when looking into the InfluxDBWriter code. Still, I'd like to see the Elasticsearch log when the status code 400 is returned. |
|
@dnsmichi Setting up a dev ES 6 and enabled debug, the log shows:
Hope that helps. |
|
Gotcha. Thanks for the heads up, I was expecting something like that. |
dnsmichi
changed the title
dnsmichi
removed
the
needs feedback
|
It is kind of a ride to bring the components together for Elastic Stack 6.x inside Puppet 4, but now I see it too with a local version of Icinga/icinga-vagrant#56. |
|
Hi, Any update on this issue, is there any workaround? Another question, I tried to send the _bulk through kibana "dev tools" and it worked, Thanks, Guy |
|
I don't think that there is a workaround, as you need to add a final "\n" to the sent body. I am working on this, still I have other projects atm. Testing a local fix. Kibana likely adds the missing newline on its own. It might be the case that Kibana developers added this in 5.x already as they probably knew about this breaking change. Two things for this issue:
|
Referencesnew line terminatorhttps://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html While researching on the matter, this newline character terminator was expected to be there in 5.x too. For some reason, the API error handling in 6.x improved and now marks this as visible error. Could be related: elastic/elasticsearch@c898e8a#diff-09b5159da8eeeb38a2773ab951cfa4ef This also mentions this: Which makes me believe that "application/json" is the current safe way to go. Strict content-typeelastic/elasticsearch@7520a10#diff-09b5159da8eeeb38a2773ab951cfa4ef also enforces a more strict content-type header. This is turned off in 5.x, but enabled in 6.x. Further reading: https://www.elastic.co/blog/strict-content-type-checking-for-elasticsearch-rest-requests - cannot be disabled in 6.x, no possible workaround. So I'd say we were using a bug in Elasticsearch 5.x which now has been fixed. There's no other way around than testing the snapshot packages once the coming PR is merged, and then wait for 2.8.1. Tests5.x
6.x
|
|
At first, I have tried running the CUrl command but with out success, assume it didn't work because this was windows version of CUrl After your response I have added the \n to my c#, when calling the rest API, and it did solved the issue. Thanks, Guy |
This enables compatibility with 6.x. This commit also fixes an incorrect HTTP response parsing which could lead into false positives. Analysis and fix in #5795 (comment) fixes #5795
This enables compatibility with 6.x. This commit also fixes an incorrect HTTP response parsing which could lead into false positives. Analysis and fix in #5795 (comment) fixes #5795 refs #5840
|
Please test the snapshot packages once available. |
|
@dnsmichi Thanks for the fix. I tested the new packege (version: v2.8.0-164-g9b75548) in my test evirenmont. A few seconds after starting or reloading the Icinga2 service the ElasticsearchWriter discontinue writing. Here my Icinga2 ElasticsearchWriter log: cat /etc/icinga2/features-enabled/elasticsearch.conf |
|
Anything interesting in the debug log? Which Elasticsearch version is used? |
|
Hi, After starting icinga2, I use
nothing is showing (only after a stop of Icinga2)
debug.log shows something after a while (like a cache or something) Using the gdb (found in Icinga2 docs)
the ElasticsearchWriter connects to elastic without any problem and without the bulk-api-error It's a fresh installation on an updated CentOS6 VM. |
|
Elasticsearch version 6.1.1. The following entries are found in my debug log: |
|
@BjoernRobbe but you'll get data in Elasticsearch? @myownhost did not test 6.1 yet, I was still at 6.0.0. |
|
@dnsmichi Sorry, I just noticed that only some sort of data arrived in elastic (version 6.0). Looking deeper shows error in elastic log:
So, I think my last post is incorrect about the difference between normal start and gdb start. Thanks |
|
@dnsmichi https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-6.0.html |
|
To be honest, I have no idea what would cause a multiple mapping. I did not see such with the events passed to 6.0 either. |
|
BTW, after deleting all indices on elastic and "normal" starting icinga2, nothing arrived in elastic, After starting icinga2 with gdb, "message(s)" arrived in elastic and log shows:
In Kibana, only one message is shown: I'm confused. |
|
Please continue with the mapping problem in #5905, this is a whole new problem compared to the bulk new line issue. |
|
@myownhost do you see similar issues in your Elasticsearch log? |
|
@dnsmichi yes i have, but i have there the same errors with icingabeat and icingabeat is working fine. Hopefully it helps:
|
|
@myownhost Ok then please continue in #5905 as well, and add your findings there too as we proceed. |
|
Reverting this for 2.8.1, as #5905 implies breaking changes for Elasticsearch 6. |
|
Ups, wrong ticket, |
Using Elasticsearch 6.0.0 gives me error code 400 on the elasticsearch feature.
My elasticsearch.conf:
Enabling debuglog shows:
Reading the docs says:
Does this feature work on Elasticsearch 6.0.0 ?
BTW, first try was with nginx in front of the elastic, but after I get the HTTP error 400, I stop the nginx and put the elastic on the lan.
My Environment
icinga2 --version): icinga2 - The Icinga 2 network monitoring daemon (version: r2.8.0-1)icinga2 feature list): api checker elasticsearch graphite ido-mysql livestatus mainlog notification perfdataThe text was updated successfully, but these errors were encountered: