Elasticsearch - Method signature of performRequest and performRequestAsync changed in ES 7.x

[renphi]

Versions used

Akka version: 2.6.5
Alpakka Verison: 2.0.0
Elasticsearch Version: 7.6.2

Expected Behavior

Can perform request against elasticsearch.

Actual Behavior

Can't perform request against elasticsearch.

ES Documentation

There is a new way of using the performRequestAsync Method (I guess it was changed in ES 6.4 - marked as depricated and dropped in ES 7.0) https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.x/java-rest-low-usage-requests.html vs. https://www.elastic.co/guide/en/elasticsearch/client/java-rest/6.3/java-rest-low-usage-requests.html

ES Breaking Changes (Version 6.4)

https://www.elastic.co/guide/en/elasticsearch/reference/6.4/breaking-changes-6.4.html#breaking_64_rest_client_changes

PR when old request methods were removed (Version 7.0)

elastic/elasticsearch@36a3e9c

Possible solution

We identify this occurrences:

alpakka/elasticsearch/src/main/scala/akka/stream/alpakka/elasticsearch/impl/ElasticsearchSourceStage.scala

Line 125 in fefc985

client.performRequestAsync(

alpakka/elasticsearch/src/main/scala/akka/stream/alpakka/elasticsearch/impl/ElasticsearchSourceStage.scala

Line 136 in fefc985

client.performRequestAsync(

alpakka/elasticsearch/src/main/scala/akka/stream/alpakka/elasticsearch/impl/ElasticsearchSimpleFlowStage.scala

Line 75 in fefc985

client.performRequestAsync(

Implement version check like used for RestBulkApi(V5|V7)

ES >=6.4 & 7.x version:
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.x/java-rest-low-usage-requests.html

val req = new Request("POST", endpoint)
        req.setOptions(RequestOptions.DEFAULT.toBuilder.addHeader("Content-Type", "application/json"))
        req.setEntity(new StringEntity(searchBody, StandardCharsets.UTF_8))
        req.addParameter("scroll", settings.scroll)
        req.addParameter("sort", "_doc")

        client.performRequestAsync(
          req,
          this
        )

ES <6.4 version:
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/6.3/java-rest-low-usage-requests.html

client.performRequestAsync(
          "POST",
          endpoint,
          Map("scroll" -> settings.scroll, "sort" -> "_doc").asJava,
          new StringEntity(searchBody, StandardCharsets.UTF_8),
          this,
          new BasicHeader("Content-Type", "application/json")
        )

[ennru]

This becomes a problem only if we change the Elasicsearch client version used by Alpakka (which by default is using 6.3.1), right?
I assume Alpakka could upgrade to a later 6.x version without breaking user code?

[mrooding]

@ennru I'd be willing to take a stab at updating the elasticsearch-rest-client since I'm having dependency conflicts when using alpakka-elasticsearch next to elastic4s (which depends on elasticsearch-rest-client:7.x.x).

Is there a reason for not moving Alpakka to elasticsearch-rest-client 7? If there is, I can submit a PR to move it to 6.4+ to fix the incompatibility with performRequestAsync

[ennru]

The reason not to upgrade to an API-breaking version is the same as for upgrading: people that use the Elasticsearch client API in their code can't easily upgrade to a later Alpakka Elasticsearch version. This is why we plan to have regular major releases of Alpakka which allow breaking updates of the used client libraries.

So I'm afraid we won't be able to merge the upgrade before we open master for Alpakka 3.0.

It would be interesting to do a stab on removing the Elasticsearch client altogether and use the Akka HTTP client instead.

[mrooding]

That makes sense. What is your roadmap for moving to frequent major releases? For now, I've got it running with my own packaged version using the PR I submitted but that's far from an ideal solution.

There are not a lot of API calls going on so I'll submit a PR to move to Akka HTTP

[ennru]

I believe two major releases per year make sense for Alpakka.

[seglo]

Not applicable now that client was was replaced with Akka Http calls in #2347.