DATAES-585 - Upgraded to Elasticsearch 7.1.0.

[raynigon]

Removed old Client Implementations, switched to RestHighLevelClient for Tests. Removed types from Annotations and Implementation.

• You have read the Spring Data contribution guidelines.
• There is a ticket in the bug tracker for the project in our JIRA.
• You use the code formatters provided here and have them applied to your changes. Don’t submit any formatting related changes.
• You submit test cases (unit or integration tests) that back your changes.
• You added yourself as author in the headers of the classes you touched. Amend the date range in the Apache license header if needed. For new types, add the license header (copy from another file and set the current year only).

[raynigon]

@sothawo This is the implementation for spring-data-elasticsearch 4.x copied from #282

[sothawo]

Thank you for the contribution, Simon. At the moment we are more focused on the master (3.2) branch, more intensive work with regards to ES7 in the 4.x branch will come after the 3.2 release which is part of Spring Data Moore.

The branch 4.x currently lags behind and needs to be updated to contain the actual changes from master. This I think should be done before merging your changes.

I just had a short look at your changes, what I noticed: You removed the TransportClient; this is too early, the TransportClient is deprecated in ES7, but still available, and so we still need to support it.

@mp911de, @christophstrobl can you give more information how work on the 4.x branch will be going on?

[mp911de]

We will drop the 4.x branch after releasing Moore GA in August so 4.x work will happen on master after the GA release. For this PR, this isn't a problem because it can be rebased to master and we can switch the target branch in PR's.

We should retain TransportClient usage as long as it is available. With 4.0, we will deprecate all classes that use TransportClient.

[pivotal-issuemaster]

@raynigon Please sign the Contributor License Agreement!

Click here to manually synchronize the status of this Pull Request.

See the FAQ for frequently asked questions.

[pivotal-issuemaster]

@raynigon Thank you for signing the Contributor License Agreement!

[yami12376]

When there will be ES 7.1 support?

[sothawo]

as Mark wrote, version 3.2 will be released with Spring Data Moore in August, from then on, the main work on master will be on ES 7

[mp911de]

We're facing a change in release dates as Spring Framework 5.2 has moved its GA date to September 4. I'm going to discuss an ES 7.x upgrade with Spring Boot and Spring Data teams.

[sothawo]

But I think we shouldn't move the 3.2.x branch to ES 7, that would leave users running ES 6 with 3.1. (which uses ES 6.2.2, and 6.5/6.7 had breaking changes)

[mp911de]

The alternative is to leave dependencies as they are and switch with Spring Data ES 4.x directly to ES 8.

Elasticsearch is moving way faster with their releases than Spring Data is able to catch up.

[sothawo]

We definitely need to support 6.6+ as AWS (and other providers probably as well) do not yet offer ES 7 and I think that the number of people running applications against these cloud-provider ES clusters is growing.

[DanielFran]

Since ES introduced free support for security in 6.8+ and 7.1+ (see https://www.elastic.co/blog/security-for-elasticsearch-is-now-free), this is important to support those versions asap otherwise 'clients' will be blocked to use those functionalities with SDE.

[sothawo]

good hint, thanks you; when supporting 7 after the release of Moore this should go to 7.2 (the latest version now).
As for the 3.2. branch, this could be updated to use 6.8.1 instead of 6.7.2, but it is as well possible to either use it with the 6.7.2 client libs to access a 6.8.1 cluster, or the version of the client libs can be set to 6.8.1 in the maven pom. Both variants work.
But at the moment I do not see why we shouldn't go to use 6.8.1 on Spring Data Elasticsearch 3.2.x

[mp911de]

We could upgrade to 6.8.x before shipping Moore RC2 next week. It’s somewhat late but having security controls in place makes the upgrade quite valueable.

[sothawo]

I created DATAES-606, I do this tomorrow or latest on thursday.

[Chris-Almond]

Hi @sothawo

Your comment above mentioning that 3.1.x won't work with ES 6.5: Does that mean ES 6.5+ is completely unusable with 3.1.x, and users of ES 6.5+ will need to wait for 3.2.x, or does it mean that a known set of features don't work?

If it's the former, is there an estimated or targeted timeline for a 3.2.x release?

Thanks for your awesome work on this! 👍

[sothawo]

@Chris-Almond, yes SDE 3.1.x is built against the client libs of ES 6.2.2 which will not work with a 6.5 cluster. And using 3.1 with explicitly updating the client libs to 6.5 doesn't work either, because there were breaking changes in ES

For example for some calls the return type was changed from concrete subtype to the abstract supertype. For the compiler, this is now problem, as there the return type does not belong to the method signature. But for the run time this is different, and then the methods are not found in the classfile. And some more things like that.

Spring Data Moore.RC2 (which for SDE is the 3.2.x) is planned for next week, GA is planned as far as I know for the beginning of september

[mp911de]

FWIW, our wiki contains the release schedule.

[Chris-Almond]

Thank you for the detailed explanation and wiki link. Very excited for the coming Moore GA (and RC)!

[NoBugBoy]

7.2什么时候支持？有7.2相关的教程了嘛

[steghio]

Hi all, just wanted to add our interest to support Elastic search 7+ for the same reason as #284 (comment), with current 3.2.0.M1 we cannot connect to Elasticsearch 7.0.1 using SSL and user authentication.

***************************
APPLICATION FAILED TO START
***************************

Description:

An attempt was made to call a method that does not exist. The attempt was made from the following location:

    org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.indexExists(ElasticsearchRestTemplate.java:736)

The following method did not exist:

    org.elasticsearch.client.IndicesClient.exists(Lorg/elasticsearch/action/admin/indices/get/GetIndexRequest;[Lorg/apache/http/Header;)Z

The method's class, org.elasticsearch.client.IndicesClient, is available from the following locations:

    jar:file:/maven/repo/org/elasticsearch/client/elasticsearch-rest-high-level-client/7.0.1/elasticsearch-rest-high-level-client-7.0.1.jar!/org/elasticsearch/client/IndicesClient.class

It was loaded from the following location:

    file:/maven/repo/org/elasticsearch/client/elasticsearch-rest-high-level-client/7.0.1/elasticsearch-rest-high-level-client-7.0.1.jar


Action:

Correct the classpath of your application so that it contains a single, compatible version of org.elasticsearch.client.IndicesClient

[DanielFran]

@sothawo Did you close this PR by mistake? This target 4.x branch (now master)...
This maybe just need a rebase

[sothawo]

We have created a 4.0.x branch so we can go on working on the changes we want to implement even before 3.2.0.GA is out and we continue to work on master; I branched 4.0.x from the 3.2.0.RC2 master, and then rebased and merged the old 4.x onto 4.0.x. So the changes from 4.x are contained in the 4.0.x, therefore I closed 4.x. So this PR was kind of unintentionally closed with the branch.

About rebasing this PR to 4.0.x: I am currently checking the changes needed for 4.0.x to use Elasticsearch 7.3 and found, that we for the time being, we will merely deprecate the TransportClient and the type attribute of the @Document annotation, but do not remove them, as these are still available in ES 7.3, and we first want to have Spring Data Elasticsearch use ES 7.3 without as few changes for the user as possible.

This PR removes quite too much stuff and would introduce more breaking changes for the user than we intend to, so we would have either needed to rework this or drop it and create a new change - which we think would have been a better solution.

I think that at the end of the week we will have a 4.0.x branch that can be used with ES 7.3.

So I'm sorry that this PR was closed without further explanation, this wasn't my intent.

[DanielFran]

Thanks for the explanation, but since they are lot of people that claim for ES 7.x support I found it was better to add some comments to this PR....

[sothawo]

Sure, you're totally right. It was never my intent to close this without explanation. And we are currently working on getting 4.0.x to use ES 7 as soon as possible.

[sothawo]

branch 4.0.x now is updated to use Elasticsearch 7.3

[Michael-LLD]

Excuse me, When 4.0.x will be released? @sothawo

[mp911de]

GA around Q2/2020, milestones likely later this year.

[adivardhan]

Hi, can we use Elasticsearch 7.X with Spring Data now?

I can only see spring-data-elasticsearch 3.X in RELEASE.

[jvmlet]

4.x is RC still

[sothawo]

https://github.com/spring-projects/spring-data-commons/wiki/Release-Train-Neumann

[adivardhan]

@sothawo Thanks for updating.
I'll have to upgrade to ES 7 and won't be able to stall till May 6th'20.
Probably use some other way.

[adivardhan]

@sothawo Looks like release is pushed from 6th May to 12th May. Hopefully, on 12th it will be released. Please let me know!

PS: My code works with snapshot release but can't go in prod with it!