Document that clients are compatible across at least 2 major versions

[dblock]

What kind of business use case are you trying to solve? What are your requirements?

Coming from #12, clients are being upgraded to support OpenSearch 2.0 while breaking compatibility with 1.0. For example, opensearch-java 1.0 works against OpenSearch 1.x, opensearch-java 2.0 works against OpenSearch 2.x, but there's no version of client that works against both OpenSearch 1.x and 2.x. Thus, in order to upgrade from OpenSearch 1.x to 2.x one needs to either run a cluster in mixed mode for a long time, or take downtime.

With rolling upgrades:

1. Upgrade some nodes to 2.0, cluster is now in mixed state.
2. Upgrade all clients to 2.0 and direct trafic from these clients to 2.0 nodes.
3. Finish upgrading the cluster to 2.0.

What is the problem? What is preventing you from meeting the requirements?

Users have to upgrade both clients and server at the same time and/or run in mixed cluster mode for a very long time. This is a lot of work, hard to coordinate, risky, etc. Most users will choose not to upgrade across major versions.

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?
Ensure that there's at least 1 version of a client that works against two major versions. For example, opensearch-java 2.0 would work against 1.x and 2.0 and deprecate all 1.x methods, and remove them in 3.0.

Describe alternatives you've considered
REST API versioning so users can specify their desired API version to ensure client compatibility is independent of server upgrades.

Other

This is tangentially related to opensearch-project/OpenSearch#3023.

[dlvenable]

Ensure that there's at least 1 version of a client that works against two major versions. For example, opensearch-java 2.0 would work against 1.x and 2.0 and deprecate all 1.x methods, and remove them in 3.0.

I think the goal should be that at least 1 version of a client in major version N works in major version N+1.

It seems it could be a problem to require that OpenSearch 2.0 client work against OpenSearch 1.0. Some breaking changes may make this infeasible.

[dlvenable]

As part of this proposal, I'd also like to include that clients be sure to correctly deprecate APIs, fields, or methods which are deprecated. This will make it easier for a user to remove all the methods which are slated for removal in the upcoming version. For Java, for example, these APIs should have the @Deprecated annotation.

[VachaShah]

The removals happening for 2.0 had already been deprecated and made optional in the 1.x versions in the client (example: https://github.com/opensearch-project/opensearch-java/blob/1.x/java-client/src/main/java/org/opensearch/client/opensearch/core/GetSourceRequest.java#L225).

[nknize]

👎
Wire compatiblity is the last Minor release of Major - 1; this is to avoid long running deprecations in the transport layer.
In core, Index compatibility is the first major release of Major - 1; this is to support rolling upgrades across the entire previous version.

Separately, the REST index API should be versioned to ensure the REST interface is compatible across all running versions. We just don't have that yet; so I'd put the effort into developing API versioning.

[reta]

As per my understanding, the REST Java client for 2.0 should have no issues working with 1.x (for existing APIs) - in theory. The major change here is, of cause, removal of _type and its usage across all HTTP endpoints. We know that 1.x supports dual with/without _type endpoints, but 2.0 - only without _type, and this is what REST Java client for 2.0 should use.

[andrross]

As per my understanding, the REST Java client for 2.0 should have no issues working with 1.x (for existing APIs)

To generalize this a bit, I would expect that a 2.0 REST client should work against 1.x assuming you're not using any new features that exist only in 2.0. Conversely, a 1.x REST client should work against a 2.0 server assuming you're not using any deprecated features that were removed in 2.0. Is that not the case?

[reta]

To generalize this a bit, I would expect that a 2.0 REST client should work against 1.x assuming you're not using any new features that exist only in 2.0

👍

Conversely, a 1.x REST client should work against a 2.0 server assuming you're not using any deprecated features that were removed in 2.0. Is that not the case?

👍 (but the _type removal affects majority of the HTTP API landscape)

[dblock]

For the Java client I opened opensearch-project/opensearch-java#152 to re-add compatibility with 1.x in the terms described above.

[dblock]

@nknize Do you not agree that one would expect that a 2.0 REST client should work against 1.x assuming you're not using any new features that exist only in 2.0?

[nknize]

Do you not agree that one would expect that a 2.0 REST client should work against 1.x assuming you're not using any new features that exist only in 2.0?

Not w/ the current implementation, no. This is an argument for REST API versioning.

Update: I caution that this proposal is an intermediate hack that will force OpenSearch to carry server side tech debt and deprecated features for an extra major version making it harder to remove stale code until REST API Versioning becomes available.

[nknize]

Hello community. If you were brought here from the RFC Tweet be sure to read the RFC Steps to make OpenSearch Extensible [RFC] REST API Versioning to get full context on a REST API Versioning. This will enable users to specify their desired API version to ensure client compatibility is independent of server upgrades while providing full compatibility across clients without requiring to carry deprecated features or tech debt across more than one major server version.

We will update the description to this accordingly