-
Notifications
You must be signed in to change notification settings - Fork 24.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Snapshot and restore and shrink do not play well together #24257
Comments
Clarification: if you run without |
Good luck! |
When an index is shrunk using the shrink APIs, the shrink operation adds some internal index settings to the shrink index, for example `index.shrink.source.name|uuid` to denote the source index, as well as `index.routing.allocation.initial_recovery._id` to denote the node on which all shards for the source index resided when the shrunken index was created. However, this presents a problem when taking a snapshot of the shrunken index and restoring it to a cluster where the initial recovery node is not present, or restoring to the same cluster where the initial recovery node is offline or decomissioned. The restore operation fails to allocate the shard in the shrunken index to a node when the initial recovery node is not present, and a restore type of recovery will *not* go through the PrimaryShardAllocator, meaning that it will not have the chance to force allocate the primary to a node in the cluster. Rather, restore initiated shard allocation goes through the BalancedShardAllocator which does not attempt to force allocate a primary. This commit fixes the aforementioned problem by not requiring allocation to occur on the initial recovery node when the recovery type is a restore of a snapshot. This commit also ensures that the internal shrink index settings are recognized and not archived (which can trip an assertion in the restore scenario). Closes elastic#24257
…24322) When an index is shrunk using the shrink APIs, the shrink operation adds some internal index settings to the shrink index, for example `index.shrink.source.name|uuid` to denote the source index, as well as `index.routing.allocation.initial_recovery._id` to denote the node on which all shards for the source index resided when the shrunken index was created. However, this presents a problem when taking a snapshot of the shrunken index and restoring it to a cluster where the initial recovery node is not present, or restoring to the same cluster where the initial recovery node is offline or decomissioned. The restore operation fails to allocate the shard in the shrunken index to a node when the initial recovery node is not present, and a restore type of recovery will *not* go through the PrimaryShardAllocator, meaning that it will not have the chance to force allocate the primary to a node in the cluster. Rather, restore initiated shard allocation goes through the BalancedShardAllocator which does not attempt to force allocate a primary. This commit fixes the aforementioned problem by not requiring allocation to occur on the initial recovery node when the recovery type is a restore of a snapshot. This commit also ensures that the internal shrink index settings are recognized and not archived (which can trip an assertion in the restore scenario). Closes #24257
…24322) When an index is shrunk using the shrink APIs, the shrink operation adds some internal index settings to the shrink index, for example `index.shrink.source.name|uuid` to denote the source index, as well as `index.routing.allocation.initial_recovery._id` to denote the node on which all shards for the source index resided when the shrunken index was created. However, this presents a problem when taking a snapshot of the shrunken index and restoring it to a cluster where the initial recovery node is not present, or restoring to the same cluster where the initial recovery node is offline or decomissioned. The restore operation fails to allocate the shard in the shrunken index to a node when the initial recovery node is not present, and a restore type of recovery will *not* go through the PrimaryShardAllocator, meaning that it will not have the chance to force allocate the primary to a node in the cluster. Rather, restore initiated shard allocation goes through the BalancedShardAllocator which does not attempt to force allocate a primary. This commit fixes the aforementioned problem by not requiring allocation to occur on the initial recovery node when the recovery type is a restore of a snapshot. This commit also ensures that the internal shrink index settings are recognized and not archived (which can trip an assertion in the restore scenario). Closes #24257
…24322) When an index is shrunk using the shrink APIs, the shrink operation adds some internal index settings to the shrink index, for example `index.shrink.source.name|uuid` to denote the source index, as well as `index.routing.allocation.initial_recovery._id` to denote the node on which all shards for the source index resided when the shrunken index was created. However, this presents a problem when taking a snapshot of the shrunken index and restoring it to a cluster where the initial recovery node is not present, or restoring to the same cluster where the initial recovery node is offline or decomissioned. The restore operation fails to allocate the shard in the shrunken index to a node when the initial recovery node is not present, and a restore type of recovery will *not* go through the PrimaryShardAllocator, meaning that it will not have the chance to force allocate the primary to a node in the cluster. Rather, restore initiated shard allocation goes through the BalancedShardAllocator which does not attempt to force allocate a primary. This commit fixes the aforementioned problem by not requiring allocation to occur on the initial recovery node when the recovery type is a restore of a snapshot. This commit also ensures that the internal shrink index settings are recognized and not archived (which can trip an assertion in the restore scenario). Closes #24257
The simplest way to reproduce locally is with
gradle run
and then:Kibana will give you "socket hang up" error because Elasticsearch has tripped an assertion and killed itself:
If you run without
-ea
then the shrunken index won't allocation properly because of theindex.routing.allocation.initial_recovery
setting. That setting cannot be cleared.The text was updated successfully, but these errors were encountered: