Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't update indexShard if it has been removed before #7509

Merged
merged 1 commit into from Aug 29, 2014

Conversation

Projects
None yet
3 participants
@s1monw
Copy link
Contributor

s1monw commented Aug 29, 2014

Today we have logic that removes a shard from the indexservice if
the shard has changed ie. from replica to primary or if it's recovery
source vanished etc. This can cause shards from been not allocated at
all on a nodes causeing delete requests to timeout since we were waiting
for shards on nodes that got dropped due to a IndexShardMissingException

you'd see exceptions like this before:

  1> org.elasticsearch.index.IndexShardMissingException: [test][5] missing
  1>    at org.elasticsearch.index.service.InternalIndexService.shardInjectorSafe(InternalIndexService.java:293)
  1>    at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:572)
  1>    at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:184)
  1>    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:444)
@bleskes

View changes

src/main/java/org/elasticsearch/indices/cluster/IndicesClusterStateService.java Outdated
@@ -548,11 +549,13 @@ private void applyNewOrUpdatedShards(final ClusterChangedEvent event) throws Ela
// if the current and global routing are initializing, but are still not the same, its a different "shard" being allocated
// for example: a shard that recovers from one node and now needs to recover to another node,
// or a replica allocated and then allocating a primary because the primary failed on another node
boolean shardHashBeenRemoved = false;

This comment has been minimized.

Copy link
@bleskes

bleskes Aug 29, 2014

Member

Typo.. shardHas_h_BeenRemoved.

@bleskes

This comment has been minimized.

Copy link
Member

bleskes commented Aug 29, 2014

LGTM. Nice catch

@s1monw s1monw added v2.0.0 labels Aug 29, 2014

@s1monw s1monw force-pushed the s1monw:fix_random_exceptions branch Aug 29, 2014

[CORE] Don't update indexShard if it has been removed before
Today we have logic that removes a shard from the indexservice if
the shard has changed ie. from replica to primary or if it's recovery
source vanished etc. This can cause shards from been not allocated at
all on a nodes causeing delete requests to timeout since we were waiting
for shards on nodes that got dropped due to a IndexShardMissingException

Closes #7509

@s1monw s1monw removed v1.2.5 labels Aug 29, 2014

@s1monw s1monw force-pushed the s1monw:fix_random_exceptions branch to 1bb0677 Aug 29, 2014

s1monw added a commit that referenced this pull request Aug 29, 2014

[CORE] Don't update indexShard if it has been removed before
Today we have logic that removes a shard from the indexservice if
the shard has changed ie. from replica to primary or if it's recovery
source vanished etc. This can cause shards from been not allocated at
all on a nodes causeing delete requests to timeout since we were waiting
for shards on nodes that got dropped due to a IndexShardMissingException

Closes #7509

@s1monw s1monw merged commit 1bb0677 into elastic:master Aug 29, 2014

@s1monw s1monw deleted the s1monw:fix_random_exceptions branch Aug 29, 2014

s1monw added a commit that referenced this pull request Sep 8, 2014

[CORE] Don't update indexShard if it has been removed before
Today we have logic that removes a shard from the indexservice if
the shard has changed ie. from replica to primary or if it's recovery
source vanished etc. This can cause shards from been not allocated at
all on a nodes causeing delete requests to timeout since we were waiting
for shards on nodes that got dropped due to a IndexShardMissingException

Closes #7509

@clintongormley clintongormley changed the title [CORE] Don't update indexShard if it has been removed before Resiliency: Don't update indexShard if it has been removed before Sep 8, 2014

@clintongormley clintongormley changed the title Resiliency: Don't update indexShard if it has been removed before Don't update indexShard if it has been removed before Jun 7, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.