Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

index.routing.allocation.initial_recovery._id split after shrink #43955

Closed
laf-rge opened this issue Jul 4, 2019 · 6 comments · Fixed by #44053

Comments

@laf-rge
Copy link

commented Jul 4, 2019

Elasticsearch version (bin/elasticsearch --version): 7.1.1

Plugins installed: []

JVM version (java -version):

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:

Steps to reproduce:

  1. Create an index
  2. Shrink an index
  3. remove the node the shrink occurred on from the cluster
  4. Wait for index to recover
  5. attempt to split the index

Provide logs (if relevant):
Split fails on allocation. Decider says all primary shards must be on the dead node. Can not remove index.routing.allocation.initial_recovery._id setting from index.

@laf-rge

This comment has been minimized.

Copy link
Author

commented Jul 4, 2019

Similar to #31787

@laf-rge

This comment has been minimized.

Copy link
Author

commented Jul 4, 2019

You might be asking yourself: why would you split a node after a shrink? Short answer: bad ILM policy.

@elasticmachine

This comment has been minimized.

Copy link
Collaborator

commented Jul 4, 2019

@henningandersen henningandersen added the >bug label Jul 4, 2019
@DaveCTurner DaveCTurner self-assigned this Jul 6, 2019
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Jul 8, 2019
If an index is the result of a shrink then it will have a value set for
`index.routing.allocation.initial_recovery._id`. If this index is subsequently
split then this value will be copied over, forcing the initial allocation of
the split shards to occur on the node on which the shrink took place. Moreover
if this node no longer exists then the split will fail.  This commit suppresses
the copying of this setting when splitting an index.

Fixes elastic#43955
@DaveCTurner

This comment has been minimized.

Copy link
Contributor

commented Jul 8, 2019

Thanks for the report @laf-rge. Splitting an index after a shrink is a legitimate thing to do so this does look wrong. I opened #44053 to address this.

pull bot pushed a commit to Pandinosaurus/elasticsearch that referenced this issue Jul 8, 2019
If an index is the result of a shrink then it will have a value set for
`index.routing.allocation.initial_recovery._id`. If this index is subsequently
split then this value will be copied over, forcing the initial allocation of
the split shards to occur on the node on which the shrink took place. Moreover
if this node no longer exists then the split will fail.  This commit suppresses
the copying of this setting when splitting an index.

Fixes elastic#43955
DaveCTurner added a commit that referenced this issue Jul 8, 2019
If an index is the result of a shrink then it will have a value set for
`index.routing.allocation.initial_recovery._id`. If this index is subsequently
split then this value will be copied over, forcing the initial allocation of
the split shards to occur on the node on which the shrink took place. Moreover
if this node no longer exists then the split will fail.  This commit suppresses
the copying of this setting when splitting an index.

Fixes #43955
@laf-rge

This comment has been minimized.

Copy link
Author

commented Jul 8, 2019

This is great. Is there a fix for the current version? Will the updated version also fix this?
edit I meant workaround.

@DaveCTurner

This comment has been minimized.

Copy link
Contributor

commented Jul 9, 2019

One possible workaround is to use reindex instead of simply splitting the index. This will take longer, unfortunately, but seems likely to work. Another idea, assuming you haven't already shrunk this index down to a single shard, is to shrink it again (which will overwrite the index.routing.allocation.initial_recovery._id setting) and then split it from there.

DaveCTurner added a commit that referenced this issue Jul 9, 2019
If an index is the result of a shrink then it will have a value set for
`index.routing.allocation.initial_recovery._id`. If this index is subsequently
split then this value will be copied over, forcing the initial allocation of
the split shards to occur on the node on which the shrink took place. Moreover
if this node no longer exists then the split will fail.  This commit suppresses
the copying of this setting when splitting an index.

Fixes #43955
DaveCTurner added a commit that referenced this issue Jul 9, 2019
If an index is the result of a shrink then it will have a value set for
`index.routing.allocation.initial_recovery._id`. If this index is subsequently
split then this value will be copied over, forcing the initial allocation of
the split shards to occur on the node on which the shrink took place. Moreover
if this node no longer exists then the split will fail.  This commit suppresses
the copying of this setting when splitting an index.

Fixes #43955
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.