Core: Upgrade API should write minimum compatible version that the index was upgrade to

[imotov]

In #11072 we are adding a check that will prevent opening of old indices. However, this check doesn't take into consideration the fact that indices can be made compatible with the current version through upgrade API. In order to make compatibility check aware of the upgrade, the upgrade API should write a new setting index.version.minimum_compatible that will indicate the minimum compatible version of elasticsearch/lucene this index is compatible with.

[rjernst]

I don't think this should just be on upgrade? It should be all merges really, since it may be true older segments are simply merged away naturally, without needing to call upgrade. I'm not sure how feasible that is though...

[rjernst]

Perhaps on every flush, since that is the "durable" part of an upgrade?

[clintongormley]

Doing a cluster state update on every flush seems excessive.

The only time this setting is used is when we move to a new major version, while having indices from 2 major versions before.

If all your ancient segments have been merged away, then calling the upgrade API with only_ancient_segments would only update this setting.

@s1monw what do you think of the naming for this setting? And what should go in it? ES version or Lucene version?

[rjernst]

Doing a cluster state update on every flush seems excessive.

It would only need to do the update if the min version changed?

[clintongormley]

true :)

[s1monw]

Perhaps on every flush, since that is the "durable" part of an upgrade?

I think we can actually do that since we only need to change the index settings / metadata iff we have a change in the min version. I think this should be rather rare?

[imotov]

Adding a setting during as part of Upgrade API is relatively simple and straightforward change. Updating setting on each flush/merge will require 1) plug a segment version check into merge process and checking the version of all segments at the end of merge, 2) sending the minimum segment version to some centralized location, probably a service running on master, 3) the said service will need to keep track of minimum segment version across all shards, which is challenge in itself because replicas don't really have identities. 4) as soon as the minimum version across all shards changes this service will update index.version.minimum_compatible setting. Can anyone see a simpler solution?

[clintongormley]

Honestly, this is a one-time thing, when you do a major upgrade. I'd go the simple route and just set the setting when called by the upgrade API