Cluster state bulk processing of a rapid deletion & creation of an index can cause usage of a wrong meta data #9489
Assignees
Labels
:Distributed/Distributed
A catch all label for anything in the Distributed Area. If you aren't sure, use this one.
resiliency
v1.4.3
v1.5.0
v2.0.0-beta1
Comments
bleskes
added a commit
to bleskes/elasticsearch
that referenced
this issue
Feb 3, 2015
…to local In big deployment ClusterState can be large. To make sure we keep reusing objects that were promoted to the Old Gen, ZenDiscovery has an optimization where it tries to reuse existing IndexMetaData object (containing among other things the mappings) from the current cluster state if they didn't change. The comparison currently uses the index name and the metadata version. This is however not enough and we should also check the index uuid. In extreme cases, where cluster state processing is slow and the index in question is deleted and recreated and these operations are batch processed together, we can use the wrong meta data if the version is also identical. This can happen if people create the index with all meta data predefined and no settings were changed. Closes elastic#9489
bleskes
added a commit
that referenced
this issue
Feb 3, 2015
…local In big deployment ClusterState can be large. To make sure we keep reusing objects that were promoted to the Old Gen, ZenDiscovery has an optimization where it tries to reuse existing IndexMetaData object (containing among other things the mappings) from the current cluster state if they didn't change. The comparison currently uses the index name and the metadata version. This is however not enough and we should also check the index uuid. In extreme cases, where cluster state processing is slow and the index in question is deleted and recreated and these operations are batch processed together, we can use the wrong meta data if the version is also identical. This can happen if people create the index with all meta data predefined and no settings were changed. Closes #9489 Closes #9541
bleskes
added a commit
that referenced
this issue
Feb 3, 2015
…local In big deployment ClusterState can be large. To make sure we keep reusing objects that were promoted to the Old Gen, ZenDiscovery has an optimization where it tries to reuse existing IndexMetaData object (containing among other things the mappings) from the current cluster state if they didn't change. The comparison currently uses the index name and the metadata version. This is however not enough and we should also check the index uuid. In extreme cases, where cluster state processing is slow and the index in question is deleted and recreated and these operations are batch processed together, we can use the wrong meta data if the version is also identical. This can happen if people create the index with all meta data predefined and no settings were changed. Closes #9489 Closes #9541
mute
pushed a commit
to mute/elasticsearch
that referenced
this issue
Jul 29, 2015
…local In big deployment ClusterState can be large. To make sure we keep reusing objects that were promoted to the Old Gen, ZenDiscovery has an optimization where it tries to reuse existing IndexMetaData object (containing among other things the mappings) from the current cluster state if they didn't change. The comparison currently uses the index name and the metadata version. This is however not enough and we should also check the index uuid. In extreme cases, where cluster state processing is slow and the index in question is deleted and recreated and these operations are batch processed together, we can use the wrong meta data if the version is also identical. This can happen if people create the index with all meta data predefined and no settings were changed. Closes elastic#9489 Closes elastic#9541
clintongormley
added
:Distributed/Distributed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When a node falls behind in processing cluster states form the master, it processes the pending updates in bulk. If these bulk contains both a deletion and creation of the same index, the node may create new shard using the UUID of the old index. This will then cause the master to ignore the shard started notifications as the left overs of the previous index (since the index was just created, they shards are empty and they come in quickly). This can leave the shard as stuck in initializing.
The text was updated successfully, but these errors were encountered: