New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Don't allow indices containing too-old segments to be opened #11072
Don't allow indices containing too-old segments to be opened #11072
Conversation
Maybe I'm misunderstanding this, but it looks like this check is only on the ES version the index was created with, and not the lucene index? While this will certainly block upgrades that are not possible, it would also fail indexes that were already upgraded (ie don't have any lucene 3.x segments)? As far as I understand, the created version is never modified (certainly not by merging or the upgrade api)? |
@rjernst great observation! Sorry, I forgot to mention this in the description. I have updated the description. |
try { | ||
indexMetaData = metaDataIndexUpgradeService.upgradeIndexMetaData(indexMetaData); | ||
} catch (Exception ex) { | ||
throw new IndexException(new Index(index), "unpgradable index", ex); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we have a better message here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kimchy I will change it to "cannot open the index due to upgrade failure" unless you have better suggestions. The reason why this index cannot be upgraded and possible remedies will be in the inner exception that will be returned as part of the response, so there is probably no reason to repeat them in this exception as well.
@imotov the code looks ok, but just so I understand how we will work in 2.0. Effectively, any version that has created version before 0.90 will require going through 1.6 for an upgrade, right? |
@kimchy correct, if a cluster contains at least one index that was created before 0.90 such cluster will have to be upgraded to v1.6 before it can be upgraded to 2.0. |
if (indexMetaData.getState() == IndexMetaData.State.OPEN && isSupportedVersion(indexMetaData) == false) { | ||
throw new IllegalStateException("The index [" + indexMetaData.getIndex() + "] was created before v0.90.0 and wasn't upgraded." | ||
+ " This index should be open using a version before " + Version.CURRENT.minimumCompatibilityVersion() | ||
+ " and upgraded using upgrade API."); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
upgrade API
-> the upgrade API
This looks pretty good, I left a few comments. |
LGTM |
Or the index can be closed, and have the setting updated manually, without upgrading to 1.6. The migration plugin https://github.com/elastic/elasticsearch-migration/ checks for minimum segment version and checks for the presense of this setting. We could add a link to instructions about how to set it without upgrading |
When index is introduced into the cluster via cluster upgrade, restore or as a dangled index the MetaDataIndexUpgradeService checks if this index can be upgraded to the current version. If upgrade is not possible, the newly upgraded cluster startup and restore process are aborted, the dangled index is imported as a closed index that cannot be open. Closes elastic#10215
dce9a38
to
21ed6bb
Compare
…ble version that the index was upgraded to In elastic#11072 we are adding a check that will prevent opening of old indices. However, this check doesn't take into consideration the fact that indices can be made compatible with the current version through upgrade API. In order to make compatibility check aware of the upgrade, the upgrade API should write a new setting `index.version.minimum_compatible` that will indicate the minimum compatible version of lucene this index is compatible with and `index.version.upgraded` that will indicate the version of elasticsearch that performed the upgrade. Closes elastic#11095
…ble version that the index was upgraded to In elastic#11072 we are adding a check that will prevent opening of old indices. However, this check doesn't take into consideration the fact that indices can be made compatible with the current version through upgrade API. In order to make compatibility check aware of the upgrade, the upgrade API should write a new setting `index.version.minimum_compatible` that will indicate the minimum compatible version of lucene this index is compatible with and `index.version.upgraded` that will indicate the version of elasticsearch that performed the upgrade. Closes elastic#11095
…ble version that the index was upgraded to In #11072 we are adding a check that will prevent opening of old indices. However, this check doesn't take into consideration the fact that indices can be made compatible with the current version through upgrade API. In order to make compatibility check aware of the upgrade, the upgrade API should write a new setting `index.version.minimum_compatible` that will indicate the minimum compatible version of lucene this index is compatible with and `index.version.upgraded` that will indicate the version of elasticsearch that performed the upgrade. Closes #11095
When an index is introduced into the cluster via cluster upgrade, restore or as a dangled index the MetaDataIndexUpgradeService checks if this index can be upgraded to the current version. If upgrade is not possible, the newly upgraded cluster startup and restore process are aborted, the dangled index is imported as a closed index that cannot be open. This change doesn't actually checks the segment versions. Instead, it indiscriminately bans all old indices created with Elasticsearch older than v0.90. In 1.6 we will add an additional logic to upgrade API that will mark upgraded indices as "upgraded" using special setting and we will modify this process to accept such indices (see #11095).
Closes #10215