Simplify delayed shard allocation #18351
Conversation
|
@bleskes can you have a look? |
| @@ -59,18 +54,13 @@ | |||
| private final AllocationService allocationService; | |||
|
|
|||
| private AtomicBoolean rerouting = new AtomicBoolean(); | |||
| private volatile long minDelaySettingAtLastSchedulingNanos = Long.MAX_VALUE; | |||
| private volatile ScheduledFuture registeredNextDelayFuture; | |||
|
|
|||
| @Inject | |||
| public RoutingService(Settings settings, ThreadPool threadPool, ClusterService clusterService, AllocationService allocationService) { | |||
| super(settings); | |||
| this.threadPool = threadPool; | |||
There was a problem hiding this comment.
we don't need the thread pool anymore
|
@bleskes I've updated the PR. Please have another look. |
| if (earlierRerouteNeeded) { | ||
| logger.info("scheduling reroute for delayed shards in [{}] ({} delayed shards)", nextDelay, | ||
| UnassignedInfo.getNumberOfDelayedUnassigned(state)); | ||
| newTask.schedule(); |
There was a problem hiding this comment.
I'm thinking some more about scheduling first and making it visible second and having second doubts on that one. Looking at the code again , why do we need to schedule first/what's the down side of doing it after setting delayedRerouteTask, so we know removeTaskAndCancel/removeIfSameTask work?
There was a problem hiding this comment.
Yes, we can do it the other way around. We know that close will be run after scheduleIfNeeded, and never the other way around.
|
Looking good! left comments here and there. Almost all very minor. |
|
@bleskes thanks for reviewing. I've updated the PR with your suggestions. Please have another look. |
|
LGTM. @ywelsch thanks for the extra iterations |
ywelsch
force-pushed
the
fix/delayed-shard-allocation
branch
from
c7eeb27
to
2369748
Compare
ywelsch
force-pushed
the
fix/delayed-shard-allocation
branch
from
2369748
to
45e8798
Compare
lcawl
added
:Distributed/Distributed
This PR simplifies the delayed shard allocation implementation by assigning clear responsibilities to the various components that are affected by delayed shard allocation:
UnassignedInfogets a boolean flagdelayedwhich determines whether assignment of the shard should be delayed. The flag gets persisted in the cluster state and is thus available across nodes, i.e. each node knows whether a shard was delayed-unassigned in a specific cluster state. Before, nodes other than the current master were unaware of that information.trueif the shard becomes unassigned due to a node leaving and the index settingindex.unassigned.node_left.delayed_timeoutbeing strictly positive. From then on, unassigned shards can only transition from delayed to non-delayed, never in the other direction.DelayedAllocationService, reacting to cluster change events, has the responsibility to schedule reroutes to remove the delay marker.Relates to #18293