Decreasing delayed allocation timeout while shard fetching can lead to longer delay

[ywelsch]

Test failure:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-intake/518/testReport/junit/org.elasticsearch.cluster.routing/DelayedAllocationIT/testDelayedAllocationChangeWithSettingTo100ms/

Scenario that explains situation:

• Due to node failing, shard allocation is delayed for one minute
• 30 seconds later, user updates delayed shard allocation to 40 second for this index -> This should allocate the shard in 10 seconds, BUT:
• While the reroute step is called during the update of the settings, shard fetching is still happening (or any other kind of reason that makes ReplicaShardAllocator call removeAndIgnore)
• This means that the setting is updated, but routing table is not (as we only update routing table if shard is delayed (which is not in this case, as the shard-fetching check comes first).
• The delay in the UnassignedInfo is still marked as 1 minute.
• RoutingService.clusterChanged checks findSmallestDelayedAllocationSettingNanos which returns 40 seconds. As this is smaller than the previous setting (which was 1 minute), it cancels existing delayed reroute, and schedules a new one (it sets minDelaySettingAtLastSchedulingNanos to 40 seconds). To determine the delay it looks at the delay stored in the shards, and only finds 1 minute delays (as UnassignedInfo was not updated), so it schedules next reroute in 1 minute (this means that the original delay is even extended by 30 seconds).
• Shard fetching is finished (2 seconds later), and a reroute is done. Here we update the delay in UnassignedInfo to 8 seconds. The routing table is now correctly updated, BUT RoutingService does not react properly to it. It compares minDelaySettingAtLastSchedulingNanos (previously set to 40 seconds with current value of findSmallestDelayedAllocationSettingNanos which still returns 40 seconds). As such it will not reschedule.
• This means that the shard will only be reallocated one minute and a half after node crashed unless the user updates delayed shard allocation for the index to a shorter time.