Content-type for Elasticsearch should be "application/x-ndjson"

[micoq]

Currently the ElasticsearchWriter use application/json for the events sent to Elasticsearch.

However, Elasticsearch defines a special content type application/x-ndjson since the content are not strictly a single JSON object but multiple JSON objects separated by newlines (the "bulk" format):
https://www.elastic.co/guide/en/elasticsearch/reference/6.0/docs-bulk.html

application/json is deprecated in Elasticsearch (since the version 5.3) and some applications which accept the bulk format (Logstash with the codec es_bulk) can drop the data after the first line (the first JSON object).

[dnsmichi]

It is not clear from Elastic whether this content-type should be the new standard, or if they ever remove application/json.

See my comment: #5795 (comment)
The deprecation issue: elastic/elasticsearch#25718
Rsyslog removed the change causing problems: rsyslog/rsyslog#1743

I'm waiting until Elastic decides where to head, until then it stays "as is". In terms of Logstash - that might break, but is not the target of this feature (first time I've heard of es_bulk).

[micoq]

I didn't tried to connect Icinga 2 directly to Elasticsearch yet, the initial idea was to use the input plugin logstash-input-http with the codec logstash-codec-es_bulk to collect checks results into a message bus (but the event REST API seems a better choice for this).

I didn't know the type application/ndjson was already defined.

Anyways, logstash-input-http can be configured to accept bulks with the content-type application/json:

http {
    ...
    additional_codecs => {"application/json" => "es_bulk"}
    ...
}

[lippserd]

I think that we should use application/x-ndjson as header for the bulk API.

[dnsmichi]

Elastic discusses the deprecation meanwhile, seems they cannot really drop it because of Kibana and apps. Anyhow, the change is just one line plus a comment, so it is done and also verified tested.