Cluster state bulk processing of a rapid deletion & creation of an index can cause usage of a wrong meta data #9489

Closed
bleskes opened this Issue Jan 29, 2015 · 0 comments

Projects

None yet

1 participant

@bleskes
Member
bleskes commented Jan 29, 2015

When a node falls behind in processing cluster states form the master, it processes the pending updates in bulk. If these bulk contains both a deletion and creation of the same index, the node may create new shard using the UUID of the old index. This will then cause the master to ignore the shard started notifications as the left overs of the previous index (since the index was just created, they shards are empty and they come in quickly). This can leave the shard as stuck in initializing.

@bleskes bleskes self-assigned this Jan 29, 2015
@bleskes bleskes added a commit to bleskes/elasticsearch that referenced this issue Feb 3, 2015
@bleskes bleskes Discovery: check in index uuid when merging incoming cluster state in…
…to local

In big deployment ClusterState can be large. To make sure we keep reusing objects that were promoted to the Old Gen, ZenDiscovery has an optimization where it tries to reuse existing IndexMetaData object (containing among other things the mappings) from the current cluster state if they didn't change. The comparison currently uses the index name and the metadata version. This is however not enough and we should also check the index uuid. In extreme cases, where cluster state processing is slow and the index in question is deleted and recreated and these operations are batch processed together, we can use the wrong meta data if the version is also identical. This can happen if people create the index with all meta data predefined and no settings were changed.

Closes #9489
397e4a2
@bleskes bleskes added a commit that referenced this issue Feb 3, 2015
@bleskes bleskes Discovery: check index uuid when merging incoming cluster state into …
…local

In big deployment ClusterState can be large. To make sure we keep reusing objects that were promoted to the Old Gen, ZenDiscovery has an optimization where it tries to reuse existing IndexMetaData object (containing among other things the mappings) from the current cluster state if they didn't change. The comparison currently uses the index name and the metadata version. This is however not enough and we should also check the index uuid. In extreme cases, where cluster state processing is slow and the index in question is deleted and recreated and these operations are batch processed together, we can use the wrong meta data if the version is also identical. This can happen if people create the index with all meta data predefined and no settings were changed.

Closes #9489
Closes #9541
c4231ce
@bleskes bleskes added a commit that referenced this issue Feb 3, 2015
@bleskes bleskes Discovery: check index uuid when merging incoming cluster state into …
…local

In big deployment ClusterState can be large. To make sure we keep reusing objects that were promoted to the Old Gen, ZenDiscovery has an optimization where it tries to reuse existing IndexMetaData object (containing among other things the mappings) from the current cluster state if they didn't change. The comparison currently uses the index name and the metadata version. This is however not enough and we should also check the index uuid. In extreme cases, where cluster state processing is slow and the index in question is deleted and recreated and these operations are batch processed together, we can use the wrong meta data if the version is also identical. This can happen if people create the index with all meta data predefined and no settings were changed.

Closes #9489
Closes #9541
2294400
@bleskes bleskes added a commit that closed this issue Feb 3, 2015
@bleskes bleskes Discovery: check index uuid when merging incoming cluster state into …
…local

In big deployment ClusterState can be large. To make sure we keep reusing objects that were promoted to the Old Gen, ZenDiscovery has an optimization where it tries to reuse existing IndexMetaData object (containing among other things the mappings) from the current cluster state if they didn't change. The comparison currently uses the index name and the metadata version. This is however not enough and we should also check the index uuid. In extreme cases, where cluster state processing is slow and the index in question is deleted and recreated and these operations are batch processed together, we can use the wrong meta data if the version is also identical. This can happen if people create the index with all meta data predefined and no settings were changed.

Closes #9489
Closes #9541
896e865
@bleskes bleskes closed this in 896e865 Feb 3, 2015
@mute mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
@bleskes bleskes Discovery: check index uuid when merging incoming cluster state into …
…local

In big deployment ClusterState can be large. To make sure we keep reusing objects that were promoted to the Old Gen, ZenDiscovery has an optimization where it tries to reuse existing IndexMetaData object (containing among other things the mappings) from the current cluster state if they didn't change. The comparison currently uses the index name and the metadata version. This is however not enough and we should also check the index uuid. In extreme cases, where cluster state processing is slow and the index in question is deleted and recreated and these operations are batch processed together, we can use the wrong meta data if the version is also identical. This can happen if people create the index with all meta data predefined and no settings were changed.

Closes #9489
Closes #9541
f98a6e3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment