Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decreasing delayed allocation timeout while shard fetching can lead to longer delay #18293

Closed
ywelsch opened this issue May 12, 2016 · 0 comments

Comments

Projects
None yet
2 participants
@ywelsch
Copy link
Contributor

commented May 12, 2016

Test failure:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-intake/518/testReport/junit/org.elasticsearch.cluster.routing/DelayedAllocationIT/testDelayedAllocationChangeWithSettingTo100ms/

Scenario that explains situation:

  • Due to node failing, shard allocation is delayed for one minute
  • 30 seconds later, user updates delayed shard allocation to 40 second for this index -> This should allocate the shard in 10 seconds, BUT:
  • While the reroute step is called during the update of the settings, shard fetching is still happening (or any other kind of reason that makes ReplicaShardAllocator call removeAndIgnore)
  • This means that the setting is updated, but routing table is not (as we only update routing table if shard is delayed (which is not in this case, as the shard-fetching check comes first).
  • The delay in the UnassignedInfo is still marked as 1 minute.
  • RoutingService.clusterChanged checks findSmallestDelayedAllocationSettingNanos which returns 40 seconds. As this is smaller than the previous setting (which was 1 minute), it cancels existing delayed reroute, and schedules a new one (it sets minDelaySettingAtLastSchedulingNanos to 40 seconds). To determine the delay it looks at the delay stored in the shards, and only finds 1 minute delays (as UnassignedInfo was not updated), so it schedules next reroute in 1 minute (this means that the original delay is even extended by 30 seconds).
  • Shard fetching is finished (2 seconds later), and a reroute is done. Here we update the delay in UnassignedInfo to 8 seconds. The routing table is now correctly updated, BUT RoutingService does not react properly to it. It compares minDelaySettingAtLastSchedulingNanos (previously set to 40 seconds with current value of findSmallestDelayedAllocationSettingNanos which still returns 40 seconds). As such it will not reschedule.
  • This means that the shard will only be reallocated one minute and a half after node crashed unless the user updates delayed shard allocation for the index to a shorter time.

@ywelsch ywelsch changed the title Decreasing delayed allocation timeout while shard fetching can lead to missing reroute Decreasing delayed allocation timeout while shard fetching can lead to longer delay May 12, 2016

@ywelsch ywelsch closed this in 31b0777 May 26, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.