Bound the shutdown drain so a stuck handler cannot stall the host

[VDBBjorn]

The problem

ControlLoop.StopAsync awaited the loop task with no bound:

if (_loop is not null) try { await _loop; } catch (OperationCanceledException) { }

A handler that ignores its CancellationToken — a blocking HTTP call, a driver that swallows the token, a lock held by another thread — therefore blocks shutdown forever. EventStoreHead.StopAsync and DeadLetterRetryLoop.StopAsync had the same shape, and RunPipelinedAsync's finally did an unbounded await Task.WhenAll(workers).

With leasing enabled this is worse than a slow exit: the replica keeps holding its processor lease while the host refuses to terminate, so LeaseAwareControlLoopGroup never releases and a blue/green handoff stalls until the orchestrator SIGKILLs the pod.

StopAsync also ignored the cancellationToken the host passed it, so even the host's own shutdown deadline had no effect.

Prompted by Marten 9.16–9.22, which added StopAndDrainTimeout for the same reason.

The change

New ControlLoopOptions.DrainTimeout (default 5 s), applied to every long-lived loop:

Loop	Bounded wait
ControlLoop.StopAsync	_loop.WaitAsync(_drainTimeout, cancellationToken)
ControlLoop.RunPipelinedAsync (worker drain)	Task.WhenAll(workers).WaitAsync(_drainTimeout)
EventStoreHead.StopAsync	_loop.WaitAsync(_drainTimeout, cancellationToken)
DeadLetterRetryLoop.StopAsync	_loop.WaitAsync(_drainTimeout, cancellationToken)

Each logs a warning on timeout (the pipelined one includes the flushed SafeCheckpoint and the number of abandoned workers). StopAsync now honours the caller's token too. The group wrappers (ControlLoopGroup, LeaseAwareControlLoopGroup, DeadLetterRetryLoopGroup) need no change — they become bounded automatically.

Why abandoning is safe

Abandoning the wait cannot lose an event. A worker that never returns also never calls watermark.MarkCompleted(), so PositionWatermark.SafeCheckpoint (_inFlight[0] - 1) stays behind its position. The final flush — which still runs, with CancellationToken.None — therefore cannot checkpoint past unprocessed work. The event is simply re-delivered on the next start. At-least-once holds exactly as before.

For the dead-letter loop, an abandoned claim is held by a time-bounded lease, so another worker picks it up once the lease expires.

Teardown of abandoned work

Two ordering hazards the naive version would have introduced, both handled:

• RunPipelinedAsync no longer uses using var pipelineCts. Abandoned workers still observe pipelineCts.Token, so disposal is deferred to a continuation that runs when they actually exit. On a clean drain it disposes inline, as before.
• ControlLoop.DisposeAsync / DeadLetterRetryLoop.DisposeAsync defer disposing the CTS (and, for ControlLoop, the processor) the same way when the loop was abandoned. ControlLoop.DisposeAsync is now idempotent.

Validation and config

DrainTimeout <= TimeSpan.Zero is rejected as ALB0004, alongside the other control-loop durations. Bound through ControlLoopOverrides (ControlLoop:DrainTimeout) and threaded through ControlLoopAssembler.Create to all four construction sites. Documented in docs/configuration.md, including how to size it against terminationGracePeriodSeconds.

Tests

New tests/Alberto.Dcb.Tests/Subscriptions/ControlLoopDrainTimeoutTests.cs (7 tests), in the style of ControlLoopPipelinedCancellationTests:

• sequential StopAsync returns within budget with a token-oblivious handler;
• an abandoned sequential handler does not advance the checkpoint;
• a stuck pipelined worker is abandoned and the checkpoint stays pinned at the last safe position (staged in two phases so it never races the loop's own progress);
• DisposeAsync after an abandoned drain neither throws nor double-tears-down, and is idempotent;
• EventStoreHead.StopAsync and DeadLetterRetryLoop.StopAsync are bounded against backends that ignore cancellation;
• the option has its default and round-trips through ControlLoopOverrides.

All gates use TaskCompletionSource, not wall-clock sleeps; every stalled fake is released in a finally so nothing outlives the test.

Local: Alberto.Dcb.Tests 1587 passed / 0 failed / 16 skipped (5 full runs), Alberto.Examples.Tests 70 passed. The new file was run 8× on its own with no flake.

Compatibility

Additive. The new constructor parameters are optional and trailing, so existing positional and named call sites compile unchanged; omitting them yields the 5 s default. PublicAPI.Unshipped.txt updated for the four new members.