How will Java clients survive the 1.x -> 2.x upgrade?

[udoprog]

I'm raising this out of concern for the up and coming 2.0 release.

Currently, java clients communicating with an elasticsearch cluster use the following dependency.

<dependency>
  <groupId>org.elasticsearch</groupId>
  <artifactId>elasticsearch</artifactId>
  <version>1.x.y or 2.x.y</version>
</dependency>

This is a very large package, essentially containing all the code necessary to operate an elasticsearch node. What clients typically are interested in is the TransportClient code.

A major version bump is allowed to carry incompatible changes, and I will be encountering several scenarios where I will have to write java clients that communicate with both 1.x and 2.x clusters.

Traditionally you have been very good at maintaining backwards compatibility, I have 1.4 clients communicating with 1.7 clusters. So Minor versions have not been that much of an issue as long as you control the feature set you use for each cluster version.

The canonical way of solving this would be to extract the public API into a separate artifact, and under a different package name (e.g. org.elasticsearch2) to allow the two different client versions to co-exist both in maven, and in the classpath.

Considering the important comment on your java-api client documentation this seems to ramp up towards becoming a major hassle for me.

Will there be some way of combating this?

[apatrida]

We also have the need to read / write to two versions at the same time that are not compatible. Have to change to REST API to do so (at least for one of them).

[adrianluisgonzalez]

I'm facing a similar issue. We use the java 1.7 transport client to talk to our 1.7 cluster. I don't see how to upgrade to 2.x without down time and upgrading all clients at the same time. Or am I missing something?

[rtijoriwala]

Is the java 2.0.0 client backwards compatible i.e. can it talk to 1.7 ES server? If so, would it work if clients are upgraded first and then server?

[adrianluisgonzalez]

@rtijoriwala, no unfortunately the 2.0.0 client is not backwards compatible if using the binary protocol based on my testing.

[udoprog]

So I'm now encountering this in trying to implement support for Elasticsearch 2.x in Heroic.

I can't just drop existing 1.x support, since I have about 16 clusters still running it. And without 2.x support I won't be able to start migrating the clusters.

I've tried to build custom relocations, but unfortunately Apache Lucene has deprecated support for version 3, and relocating Lucene versions is an absolute PITA to the degree that I can't figure out how to do it after several hours of work (loading Codecs, etc... through reflection requires consistent paths).

My last alternative is to fork the entire codebase until I've upgraded all the clusters. But it seems backwards of me to provide two different versions of my service depending on which version of a particular sub-component it can use. If every dependency required this it would be a combinatoric nightmare.

Are there any work being done on this front?

[nik9000]

Yes. We are building a rest based Java client which should be as tolerant
of the upgrade cycle as the other rest clients. That is to say: I expect
you'll still have to have some branch statements but it ought to be
manageable. I expect it'll be some time before the rest based Java client
is as nice as the transport client but I'm confident that time will come.
On May 24, 2016 8:32 PM, "John-John Tedro" notifications@github.com wrote:

So I'm now encountering this in trying to implement support for
Elasticsearch 2.x in Heroic https://github.com/spotify/heroic.

I can't just drop existing 1.x support, since I have about 16 clusters
still running it. And without 2.x support I won't be able to start
migrating the clusters.

I've tried to build custom relocations, but unfortunately Apache Lucene
has deprecated support for version 3, and relocating Lucene versions is an
absolute PITA to the degree that I can't figure out how to do it after
several hours of work (loading Codecs, etc... through reflection requires
consistent paths).

My last alternative is to fork the entire codebase until I've upgraded all
the clusters. But it seems backwards of me to provide two different
versions of my service depending on which version of a particular
sub-component it can use. If every dependency required this it would be a
combinatoric nightmare.

Are there any work being done on this front?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#13273 (comment)

[s1monw]

If you are using the Java API you are considered to be a full blown part of the cluster. Since major version require full-cluster restarts you have to restart the client as well. We all know that this sucks and we are working on a real java client since today we don't have one. The java client is really an elasticsaerch node (lightweight) even if you use the transport "client". What others have build is a custom REST interface tailored to their domain such that they can talk to different version of the cluster etc. I am not saying that is what you should do but there is no other way than replacing the java client with the new version and the new java client we are working on is not going to help for 1.x to 2.x - or lets say very very unlikely

[udoprog]

@s1monw the only request I'd like to make is that it should be possible for two major versions of the future client to co-exist in the same VM.

I know of two ways to accomplishing this:
1) The best way would be if distinct packages and/or client entry-points were used for each major version. E.g. org.elasticsearch.client.RestClient1, and org.elasticsearch.client.RestClient2. Both would need to have all their dependencies and classes located in isolation as well, like org.elasticsearch.client10 and org.elasticsearch.client20.
2) Spend a tremendous amount of effort and painstaking attention to detail to build a backwards compatible and a granular, versioned client. Like Datastax has done for Cassandra. I dislike this approach for complex APIs (like Elasticsearch) because it necessitates a lot of version management.

[s1monw]

@s1monw the only request I'd like to make is that it should be possible for two major versions of the future client to co-exist in the same VM.

the java clinet as it exists today will be removed. We will have a client that talks to the rest interface which works cross major versions.

[rtijoriwala]

@s1monw - any idea when will this new client be available?

[brusic]

@rtijoriwala You can follow the progress here: #18735

[cnwarden]

@s1monw , we have this same upgrade issue, from 1.7.3 to 2.3.2, now we are using the 1.7.3 Java API with TransportClient, does it mean we must change API to REST, then it can cross the major version? is there any other way?

[cnwarden]

@nik9000 , may I know whether we can use it now?

[clintongormley]

You can certainly start using it when 5.0 is out. Note: it is very early days and initially won't provide the IDE assistance that you get from the transport client, but it will allow you to talk to clusters of other major versions (barring any changes that may have happened to the REST API in between versions)

Given that the REST client is where we will be focussing our attention going forwards, I'm going to close this issue