Newer Elasticsearch Version deprecate the REST High Level Client in favour of the Java API Client

[rzo1]

The High Level Rest Client is deprecated in favor of the Elasticsearch Java API Client

This will affect SC, if we want to upgrade Elasticsearch from 7.5.2 to 7.17.0

We should investigate whether the current High Level REST client can be easily replaced by the new Java API client.

A migration guide is available here

[rzo1]

@jnioche would have labeld it accordingly, but I do not have the permissions.

[jnioche]

Elasticsearch 8.0 is approaching. We could either port the whole module to 8 and replace the client at the same time or maybe have a new module elasticsearch-8 and keep the existing one with HLRC for a bit? We could clean things up a bit at the same time e.g. remove the CollapsingSpout

[rzo1]

The Elasticsearch Java client is forward compatible; meaning that the client supports communicating with greater or equal minor versions of Elasticsearch. Elasticsearch language clients are only backwards compatible with default distributions and without guarantees made.

I like the elasticsearch-8 approach as users might have deployed older elasticsearch versions and the HLRC replacement might have an impact, so perhaps we can still update to 7.17.0 (and live with the deprecation warning) and later come up with a elasticsearch-8 module.

[jnioche]

The Elasticsearch Java client is forward compatible; meaning that the client supports communicating with greater or equal minor versions of Elasticsearch. Elasticsearch language clients are only backwards compatible with default distributions and without guarantees made.

I like the elasticsearch-8 approach as users might have deployed older elasticsearch versions and the HLRC replacement might have an impact, so perhaps we can still update to 7.17.0 (and live with the deprecation warning) and later come up with a elasticsearch-8 module.

I know quite a few users for whom this would be the case

[jnioche]

Elasticsearch 8 is out!
https://www.elastic.co/blog/whats-new-elastic-8-0-0

[rzo1]

https://www.elastic.co/guide/en/elasticsearch/reference/8.0/migrating-8.0.html

[ngramp]

Elasticsearch 8 is out! https://www.elastic.co/blog/whats-new-elastic-8-0-0

This related to that? The ES_IndexInit.sh script fails in multiple places. Can't delete metrics* or content* wildcard references, can't use metric* index patterns as a legacy template as it's used by Elastic Agent as per:
https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html
Can disable the builtin templates using stack.templates.enabled, but seems like it would break other features. Seems it needs to be composable, _index_template.

Also, since security is the default now, all the protocols should be changed to https as standard and method to deal with cacerts for the connections implemented (does not even seem to respect systemwide certs).

[jnioche]

thanks @devlperry
we could have an alternative version of the script in the current module to fix the issues you mentioned when running ES8
any chance you could contribute a PR for this?

[jnioche]

Note to self: BulkIngester is what BulkProcessor used to be

https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/indexing-bulk.html

[rzo1]

We dropped ES, so closing this issue is ok now.