Support elasticsearch data_streams

[thaarbach]

With Elasticsearch 7.9.x data streams was introduced.
https://www.elastic.co/guide/en/elasticsearch/reference/master/set-up-a-data-stream.html

https://www.elastic.co/guide/en/elasticsearch/reference/master/use-a-data-stream.html

Unfortunally, appender doesn't supports the bulk api in combination with data_streams. data_streams only supports create and not index.

Eg.

PUT /my-data-stream/_bulk?refresh
{"create":{ }}
{ "@timestamp": "2099-03-08T11:04:05.000Z", "user": { "id": "vlb44hny" }, "message": "Login attempt failed" }
{"create":{ }}
{ "@timestamp": "2099-03-08T11:06:07.000Z", "user": { "id": "8a4f500d" }, "message": "Login successful" }
{"create":{ }}
{ "@timestamp": "2099-03-09T11:07:08.000Z", "user": { "id": "l7gk7f82" }, "message": "Logout successful" }

Futher informations
https://www.elastic.co/de/blog/an-introduction-to-the-elastic-data-stream-naming-scheme
https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams.html

Benefit:
Index-Templates, ILM are provided by elasticsearch out of the box.

[rfoltyns]

@thaarbach Should be possible in 1.6. You missed it by 4 days - I would have included it in 1.5..

Index-Templates, ILM are provided by elasticsearch out of the box.

I'm afraid it won't be that simple, but I'll play around with it and see what can be done.

[thaarbach]

@rfoltyns
At friday i implemented ECS and it works fine. Thx for that. Then i played arround and found :

After i played arround, i realized that the bulk request are not in the correct from. But data stream and the binding to the provied ilm was created.

With this knowing, i copied the requests from failover.log and changed them to data stream conform form and put them with curl and voila the log entries appears in the data stream and log app.

curl -X PUT "localhost:9200/my-data-stream/_bulk?refresh&pretty" -H 'Content-Type: application/json' -d'
{"create":{ }}
{ "@timestamp": "2099-03-08T11:04:05.000Z", "user": { "id": "vlb44hny" }, "message": "Login attempt failed" }
{"create":{ }}
{ "@timestamp": "2099-03-08T11:06:07.000Z", "user": { "id": "8a4f500d" }, "message": "Login successful" }
{"create":{ }}
{ "@timestamp": "2099-03-09T11:07:08.000Z", "user": { "id": "l7gk7f82" }, "message": "Logout successful" }
'

Don't know how simple it is to implement them, because before each log entry you need to set the '{"created":{}}.

Thx anyway

[rfoltyns]

I got it to work yesterday with a few quick hacks. It most likely will not be the final implementation. I can push it to a separate branch later if you're ok with building it on your own.

[thaarbach]

@rfoltyns
Yes please, I just don't know when I'll get to try it. At the moment I am using a traditional index.

Have you already played with elastic apm? I like the log correlation with the correspondending traces and/or transactions.

[rfoltyns]

I just pushed those quick hacks here.

It works with Data Streams only.

Once built, run it with:

mvn clean install -pl log4j2-elasticsearch-hc -Dtest=SmokeTest#programmaticConfigTest -Dlog4j2.level=INFO

I haven't tested the XML config yet, but some flags and Log4j2 annotations are already in place.

As for APM, similar correlation should be possible already with VirtualProperty and $${ctx:correlationId}.

[rfoltyns]

Released in 1.6.0