Fix flaky RunCommand_DetachedChild_WhenSignaledBeforeReadiness test timeout - #18702
Conversation
Replace the FakeTimeProvider-based Task.Delay in the test callback with a test-controlled TaskCompletionSource. This eliminates potential reentrancy issues where FakeTimeProvider.Advance() fires a timer whose async continuation chain disposes another timer on the same provider, which can deadlock intermittently under thread pool contention on CI. Closes #18677 Co-authored-by: JamesNK <303201+JamesNK@users.noreply.github.com>
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 18702Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 18702" |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Pull request overview
This PR fixes an intermittently-timing-out test, RunCommand_DetachedChild_WhenSignaledBeforeReadiness_AwaitsAppHostTeardownBeforeExit. The test simulates an AppHost that receives a termination signal before its backchannel is established and asserts the detached CLI child waits for AppHost teardown before exiting. The previous implementation simulated teardown time with Task.Delay(..., timeProvider) and drove it via FakeTimeProvider.Advance(), which could deadlock: advancing the fake clock fires a timer whose continuation transitively disposes another timer on the same provider (the finally-block WaitAsync timeout), and inline continuations under thread-pool contention could reenter the provider's internal lock.
Changes:
- Replaced the
Task.Delay(timeProvider)teardown simulation with a test-controlledTaskCompletionSource(teardownCanFinish) gate. - Removed the
timeProvider.Advance(...)calls that drove the timer, making the assertion that the child does not exit mid-teardown fully deterministic. - Updated surrounding comments to describe the new gating mechanism.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Tests selector (audit mode)The full test matrix and all jobs still run in audit mode. The tests and jobs below are what selective CI would run under enforcement. 1 / 99 test projects · 2 jobs, from 1 changed file. Selected test projects (1 / 99)
Selected jobs (2)
How these were chosen — grouped by what changed🧪 Job reasons
Selection computed for commit |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Ankit Jain (radical)
left a comment
There was a problem hiding this comment.
[automated] ## Validation: reproduced the flake without the fix, confirmed the fix makes it deterministic
Independently validated this fix by reproducing the deadlock mechanism, confirming the fix eliminates it, and reviewing the diff.
Root cause
The pre-fix test drives teardown through fake time:
teardownStarted.TrySetResult();
await Task.Delay(RunCommand.s_gracefulShutdownBudget + TimeSpan.FromSeconds(1), timeProvider, ...); // callback
...
timeProvider.Advance(s_gracefulShutdownBudget + 100ms); // driver
timeProvider.Advance(TimeSpan.FromSeconds(1));teardownStarted is a RunContinuationsAsynchronously TCS, so its continuation — the Task.Delay(timeProvider) that registers the fake timer — is posted to the thread pool. Under pool contention the driver's Advance() calls can execute before that timer registers. The timer then anchors to the already-advanced clock, is scheduled in the future, and never fires — pendingCommand hangs and await pendingCommand.DefaultTimeout() throws TimeoutException at RunCommandTests.cs:469, which is exactly the signature in #18677. Rare because it needs a specific cross-thread interleaving (only 2 known CI occurrences).
Reproduced without the fix
Standalone harness replaying both the pre-fix and post-fix patterns against Microsoft.Extensions.TimeProvider.Testing 10.6.0 under a starved thread pool (min=1, max=2, plus background load):
| Pattern | Iterations | Deadlocks |
|---|---|---|
pre-fix (Task.Delay(tp) + Advance) |
1500 | 2 (~0.13%) |
| post-fix (this PR) | 1500 | 0 |
Deadlock signature: runTask.IsCompleted=False after advancing past the timer — the fake timer never fired, matching the CI hang.
Confirmed the fix
- Harness post-fix pattern: 0 / 1500 deadlocks under identical contention.
- Real fixed test: builds, passes, 20 / 20 repeat runs locally.
Why it's deterministic
Fake time is never advanced, so no timer ever fires → no registration-ordering race and no WakeWaiters reentrancy. The Assert.False(pendingCommand.IsCompleted) is now a true invariant rather than a timing gamble: the callback is blocked on teardownCanFinish, and RunCommand's finally awaits WaitAsync(s_detachedAppHostTeardownTimeout, _timeProvider) on a frozen clock, so pendingCommand cannot complete until teardownCanFinish.SetResult(). Synchronization reduces to well-defined TCS completion.
LGTM.
|
✅ No documentation update needed. docs_optional → test_only: No documentation update needed. No signals triggered (signal_count = 0). The only changed file is |
Description
RunCommand_DetachedChild_WhenSignaledBeforeReadiness_AwaitsAppHostTeardownBeforeExittimes out intermittently on CI due to aFakeTimeProviderreentrancy deadlock.Root cause: The test callback used
await Task.Delay(duration, timeProvider)to simulate teardown time. WhentimeProvider.Advance()fires that timer under its internal lock, the async continuation chain can transitively attempt to dispose another timer on the same provider (theWaitAsync(8s, timeProvider)timeout in RunCommand's finally block), deadlocking when inline continuations execute under thread pool contention.Fix: Replace the
Task.Delay(timeProvider)in the callback with a test-controlledTaskCompletionSource, eliminating all FakeTimeProvider reentrancy while preserving the same behavioral guarantee (CLI waits for AppHost teardown before exiting).Checklist
<remarks />and<code />elements on your triple slash comments?