Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Internal index unrecoverable when upgrade from 5.6.x fails to reindex #39339

Closed
bizybot opened this issue Feb 25, 2019 · 1 comment · Fixed by #39340
Closed

Internal index unrecoverable when upgrade from 5.6.x fails to reindex #39339

bizybot opened this issue Feb 25, 2019 · 1 comment · Fixed by #39340
Assignees
@bizybot
Labels
>bug :Data Management/Indices APIs APIs to create and manage indices and templates >upgrade

Comments

@bizybot
Copy link
Contributor

bizybot commented Feb 25, 2019 •
edited
Loading

When following the steps mentioned in the upgrade guide,
if we disable the cluster shard allocation but fail to enable it after
upgrading the nodes and plugins, the next step of upgrading internal
indices fails. As we did not check the bulk response for reindexing,
we delete the old index assuming it has been reindexed successfully.
This is fatal as we cannot recover from this state since index to be upgraded
has been deleted.

Steps to follow:-
Following the upgrade guide:
https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html

If we miss the step of enabling cluster shard allocation, during reindex operation UnavailableShardsException is thrown.
But as we do not check the BulkByScrollResponse if there were any failures and proceed as a successful operation thereby deleting the internal index.

Solution:-
There should be a pre-upgrade check before proceeding with the upgrade. Also, in the case where reindex fails for some reason, it should not delete the original index.
@bizybot bizybot self-assigned this Feb 25, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

bizybot pushed a commit to bizybot/elasticsearch that referenced this issue Feb 25, 2019
Add pre-upgrade check to test cluster routing allocation is enabled
b9d5b7d 
When following the steps mentioned in upgrade guide
https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html
if we disable the cluster shard allocation but fail to enable it after
upgrading the nodes and plugins, the next step of upgrading internal
indices fails. As we did not check the bulk response for reindexing,
we delete the old index assuming it has been created. This is fatal
as we cannot recover from this state.

This commit adds a pre-upgrade check to test the cluster shard
allocation setting and fail upgrade if it is disabled. In case there
are search or bulk failures then we remove the read only block and
fail the upgrade index request.

Closes elastic#39339
@bizybot bizybot changed the title Index unrecoverable when upgrade from 5.6.x fails on reindex error Internal Index unrecoverable when upgrade from 5.6.x fails to reindex Feb 25, 2019
@bizybot bizybot changed the title Internal Index unrecoverable when upgrade from 5.6.x fails to reindex Internal index unrecoverable when upgrade from 5.6.x fails to reindex Feb 25, 2019
@bizybot bizybot mentioned this issue Feb 25, 2019
Merged
bizybot added a commit to bizybot/elasticsearch that referenced this issue Mar 8, 2019
@bizybot
Add pre-upgrade check to test cluster routing allocation is enabled (e…
bae7e71 
…lastic#39340)

When following the steps mentioned in upgrade guide
https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html
if we disable the cluster shard allocation but fail to enable it after
upgrading the nodes and plugins, the next step of upgrading internal
indices fails. As we did not check the bulk request response for reindexing,
we delete the old index assuming it has been created. This is fatal
as we cannot recover from this state.

This commit adds a pre-upgrade check to test the cluster shard
allocation setting and fail upgrade if it is disabled. In case there
are search or bulk failures then we remove the read-only block and
fail the upgrade index request.

Closes elastic#39339
bizybot added a commit to bizybot/elasticsearch that referenced this issue Mar 8, 2019
@bizybot
Add pre-upgrade check to test cluster routing allocation is enabled (e…
75a1442 
…lastic#39340)

When following the steps mentioned in upgrade guide
https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html
if we disable the cluster shard allocation but fail to enable it after
upgrading the nodes and plugins, the next step of upgrading internal
indices fails. As we did not check the bulk request response for reindexing,
we delete the old index assuming it has been created. This is fatal
as we cannot recover from this state.

This commit adds a pre-upgrade check to test the cluster shard
allocation setting and fail upgrade if it is disabled. In case there
are search or bulk failures then we remove the read-only block and
fail the upgrade index request.

Closes elastic#39339
@bizybot bizybot mentioned this issue Mar 8, 2019
Merged
bizybot added a commit to bizybot/elasticsearch that referenced this issue Mar 8, 2019
@bizybot
Add pre-upgrade check to test cluster routing allocation is enabled (e…
0beafa9 
…lastic#39340)

When following the steps mentioned in upgrade guide
https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html
if we disable the cluster shard allocation but fail to enable it after
upgrading the nodes and plugins, the next step of upgrading internal
indices fails. As we did not check the bulk request response for reindexing,
we delete the old index assuming it has been created. This is fatal
as we cannot recover from this state.

This commit adds a pre-upgrade check to test the cluster shard
allocation setting and fail upgrade if it is disabled. In case there
are search or bulk failures then we remove the read-only block and
fail the upgrade index request.

Closes elastic#39339
@bizybot bizybot mentioned this issue Mar 8, 2019
Merged
bizybot added a commit to bizybot/elasticsearch that referenced this issue Mar 8, 2019
@bizybot
Add pre-upgrade check to test cluster routing allocation is enabled (e…
b190864 
…lastic#39340)

When following the steps mentioned in upgrade guide
https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html
if we disable the cluster shard allocation but fail to enable it after
upgrading the nodes and plugins, the next step of upgrading internal
indices fails. As we did not check the bulk request response for reindexing,
we delete the old index assuming it has been created. This is fatal
as we cannot recover from this state.

This commit adds a pre-upgrade check to test the cluster shard
allocation setting and fail upgrade if it is disabled. In case there
are search or bulk failures then we remove the read-only block and
fail the upgrade index request.

Closes elastic#39339
@bizybot bizybot mentioned this issue Mar 8, 2019
Merged
bizybot added a commit that referenced this issue Mar 12, 2019
@bizybot
Add pre-upgrade check to test cluster routing allocation is enabled (#…
db04288 
…39340) (#39815)

When following the steps mentioned in upgrade guide
https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html
if we disable the cluster shard allocation but fail to enable it after
upgrading the nodes and plugins, the next step of upgrading internal
indices fails. As we did not check the bulk request response for reindexing,
we delete the old index assuming it has been created. This is fatal
as we cannot recover from this state.

This commit adds a pre-upgrade check to test the cluster shard
allocation setting and fail upgrade if it is disabled. In case there
are search or bulk failures then we remove the read-only block and
fail the upgrade index request.

Closes #39339
bizybot added a commit that referenced this issue Mar 12, 2019
@bizybot
Add pre-upgrade check to test cluster routing allocation is enabled (#…
6b13189 
…39340) (#39816)

When following the steps mentioned in upgrade guide
https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html
if we disable the cluster shard allocation but fail to enable it after
upgrading the nodes and plugins, the next step of upgrading internal
indices fails. As we did not check the bulk request response for reindexing,
we delete the old index assuming it has been created. This is fatal
as we cannot recover from this state.

This commit adds a pre-upgrade check to test the cluster shard
allocation setting and fail upgrade if it is disabled. In case there
are search or bulk failures then we remove the read-only block and
fail the upgrade index request.

Closes #39339
bizybot added a commit that referenced this issue Mar 12, 2019
@bizybot
Add pre-upgrade check to test cluster routing allocation is enabled (#…
20c6be9 
…39340) (#39817)

When following the steps mentioned in upgrade guide
https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html
if we disable the cluster shard allocation but fail to enable it after
upgrading the nodes and plugins, the next step of upgrading internal
indices fails. As we did not check the bulk request response for reindexing,
we delete the old index assuming it has been created. This is fatal
as we cannot recover from this state.

This commit adds a pre-upgrade check to test the cluster shard
allocation setting and fail upgrade if it is disabled. In case there
are search or bulk failures then we remove the read-only block and
fail the upgrade index request.

Closes #39339
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bizybot bizybot

Labels
>bug :Data Management/Indices APIs APIs to create and manage indices and templates >upgrade
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add pre-upgrade check to test cluster routing allocation is enabled bizybot/elasticsearch
2 participants
@bizybot @elasticmachine