XContent format Versioning

[colings86]

We have the concept of versioning on the transport wire format which allows us to make changes to the product whilst maintaining backwards compatibility with nodes on previous versions (but still on the same major version). We currently have no such versioning for the REST request/response format. For the request format we tend to add backward compatibility support in the Parsers to parse the old parameters as well as the new ones (these tend to be jumbled together with the new parameters with no clear indication they are deprecated formats still there for bwc), but for the REST Response format we don't have any way of knowing what version of the format the client expects so cannot maintain bwc with changes.

We could add a query parameter to every REST action called format whose value would specify which version the server should expect for request formats and which version the client expects for response formats. This would allow us to have the same kind of bwc logic that we have for the transport layer (if the format parameter is missing then it will default of the server's version or possibly to trying all available formats).

We could then also have a new API which would return the XContentTypes and format versions supported by the server. Clients could then use this API to determine whether they are compatible with the server on client initialisation and work out which version they need to specify in the format parameter.

This would be a big and long term change and certainly would not happen overnight as we would have to slowly introduce the bwc code in the XContent parsers and renderers.

[jpountz]

I agree we need such a thing if we want to be able to change the response formats across major versions.

[nik9000]

We could add a query parameter to every REST action called format whose value would specify which version the server should expect for request formats and which version the client expects for response formats.

MediaWiki uses formatversion for something similar to this. It feels to me like exactly the kind of this that accept headers were built for though. Either way I'm +1 on the concept .

[colings86]

We discussed this in FixItFriday and we decided that for now, as breaking changes in the REST API response are rare we will support a legacy flag in the query parameters which indicates which feature should respond in the legacy format e.g. ?legacy=aggs,completion. The feature can then check in the toXContent() for the legacy flag if it has a legacy format and then respond appropriately

[rjernst]

While I understand the need, I am concerned with the semantics. When do we ever remove the "legacy" format if we dont know what version(s) that format was associated with? What if there are two breaking changes that occur "close" together (say within a major release)? While that may be unlikely, not accounting for it could be a real problem if it ever did happen. Unless what we mean is "use the previous major version format" which assumes we do not break the rest api response format within a major version (which I think we already try to follow that rule)?

[rjernst]

Also, why can it not be as simple as a boolean? I think if you want the "legacy" format (whatever that means) the entire response should be in that format, not just pieces?

[clintongormley]

Unless what we mean is "use the previous major version format" which assumes we do not break the rest api response format within a major version (which I think we already try to follow that rule)?

Yes, exactly.

I think if you want the "legacy" format (whatever that means) the entire response should be in that format, not just pieces?

Maybe, not sure... We're not promising that the response syntax will be exactly the same as the previous major version, just that we'll revert a major breaking change. Also, it may be that somebody migrates their queries, but hasn't had time to do aggs yet. not sure

[nik9000]

@colings86 is this still a thing you think we should do? We've done fairly well holding breaking changes until the next major release but I expect there are major changes that we might want to do that'd be quite difficult without this.

[colings86]

@nik9000 yes it's something I still think we should do. It's true that we have been (and still are) able to hold breaking changes to the request side of the REST API for major versions and have been able to deprecate old API parameters there etc. But the impetus for this change for me was actually more on the response side where we currently have no way of making any breaking changes in a user friendly way because there is no way for the user to transition to a new response format before upgrading to the version where the breaking change is made. For example see #17652. This change would need to move sub aggregations to their own part of the aggregation response, but there is no way for us to make this change at the moment because we have no way for the client to specify whether they need to read the old or the new format of thee response