Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't update indexShard if it has been removed before #7509

Merged
merged 1 commit into from Aug 29, 2014

Conversation

s1monw
Copy link
Contributor

@s1monw s1monw commented Aug 29, 2014

Today we have logic that removes a shard from the indexservice if
the shard has changed ie. from replica to primary or if it's recovery
source vanished etc. This can cause shards from been not allocated at
all on a nodes causeing delete requests to timeout since we were waiting
for shards on nodes that got dropped due to a IndexShardMissingException

you'd see exceptions like this before:

  1> org.elasticsearch.index.IndexShardMissingException: [test][5] missing
  1>    at org.elasticsearch.index.service.InternalIndexService.shardInjectorSafe(InternalIndexService.java:293)
  1>    at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:572)
  1>    at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:184)
  1>    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:444)

@@ -548,11 +549,13 @@ private void applyNewOrUpdatedShards(final ClusterChangedEvent event) throws Ela
// if the current and global routing are initializing, but are still not the same, its a different "shard" being allocated
// for example: a shard that recovers from one node and now needs to recover to another node,
// or a replica allocated and then allocating a primary because the primary failed on another node
boolean shardHashBeenRemoved = false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo.. shardHas_h_BeenRemoved.

@bleskes
Copy link
Contributor

bleskes commented Aug 29, 2014

LGTM. Nice catch

Today we have logic that removes a shard from the indexservice if
the shard has changed ie. from replica to primary or if it's recovery
source vanished etc. This can cause shards from been not allocated at
all on a nodes causeing delete requests to timeout since we were waiting
for shards on nodes that got dropped due to a IndexShardMissingException

Closes elastic#7509
s1monw added a commit that referenced this pull request Aug 29, 2014
Today we have logic that removes a shard from the indexservice if
the shard has changed ie. from replica to primary or if it's recovery
source vanished etc. This can cause shards from been not allocated at
all on a nodes causeing delete requests to timeout since we were waiting
for shards on nodes that got dropped due to a IndexShardMissingException

Closes #7509
@s1monw s1monw merged commit 1bb0677 into elastic:master Aug 29, 2014
@s1monw s1monw deleted the fix_random_exceptions branch August 29, 2014 13:28
s1monw added a commit that referenced this pull request Sep 8, 2014
Today we have logic that removes a shard from the indexservice if
the shard has changed ie. from replica to primary or if it's recovery
source vanished etc. This can cause shards from been not allocated at
all on a nodes causeing delete requests to timeout since we were waiting
for shards on nodes that got dropped due to a IndexShardMissingException

Closes #7509
@clintongormley clintongormley changed the title [CORE] Don't update indexShard if it has been removed before Resiliency: Don't update indexShard if it has been removed before Sep 8, 2014
@clintongormley clintongormley changed the title Resiliency: Don't update indexShard if it has been removed before Don't update indexShard if it has been removed before Jun 7, 2015
@clintongormley clintongormley added the :Distributed/Recovery Anything around constructing a new shard, either from a local or a remote source. label Jun 7, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :Distributed/Recovery Anything around constructing a new shard, either from a local or a remote source. resiliency v1.4.0.Beta1 v2.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants