Stop leaking orphaned aspire-managed NuGet search helpers

[Rafał Araszkiewicz (Arasz)]

Description

Fixes #18948. Same root cause as #18779.

aspire ls and aspire ps spawn an aspire-managed nuget search helper that never exits. They accumulate — 19 over six days in the original report, 47 overnight on my machine — each holding lock files under $TMPDIR/NuGetScratch/lock, until an unrelated dotnet restore deadlocks at "Determining projects to restore...". Repro steps are in #18948.

Three things combine.

The prefetch is never awaited. NuGetPackagePrefetcher.ExecuteAsync fired both prefetches as discarded Task.Run (L26, L55), so it returned immediately, BackgroundService.StopAsync awaited an already-completed ExecuteTask, and the CLI exited with the search still running. The log in #18948 shows it: the helper is launched and, unlike every other subprocess in that log, never gets an exit entry. Fixed by collecting both tasks and awaiting them.

The teardown path was already correct — nothing ever ran it. The token flows unbroken to LayoutProcessRunner.cs#L58, where cancelling WaitForExitAsync unwinds into the await using and ProcessExecution.cs#L533 kills the tree. Cancellation already reaps the helper. Nothing was cancelling it.

Read-only commands opted in by default. ls and ps don't implement IPackageMetaPrefetchingCommand, so they hit the default at L108 — prefetch for everything except run/publish/deploy/do. Neither uses template metadata, and neither overrides UpdateNotificationsEnabled (default false), so neither reaches the notification site at BaseCommand.cs:202 and the CLI-package search is wasted too. Both now implement the interface with both flags false, matching NewCommand, McpInitCommand and AgentInitCommand.

One second wasn't long enough to learn the command. WaitForCommandSelectionAsync gave up after 1000 ms and fell back to the null default, which enables both prefetches. Selection happens when the command's action runs (BaseCommand.cs:61), and the first-run banner plays before that, spending 1660 ms in fixed delays — so the opt-out above was bypassable on a first run or with --banner. It now waits until shutdown. Caught by Copilot review; arithmetic here.

Why the existing protection misses all this: on non-Windows the cooperative parent-liveness watchdog is the sole mechanism (LayoutProcessRunner.cs#L38-L49), and the leaked helpers sit in state T. A stopped process is never scheduled, so it cannot notice its parent died, and SIGTERM is not delivered to it — only SIGKILL clears them. #18566 tightened several leak paths, but not this one.

What this does not guarantee

Release builds cap host shutdown at 200 ms (Program.cs#L317-L323). Cancel → kill → drain normally fits inside that, but if it overruns, the helper is still abandoned. Strictly better than today, where nothing cancels it at all — not a hard guarantee. I can add an explicit shutdown step with its own budget if you want one.

One behaviour change to rule on

Waiting for selection instead of guessing means invocations where no command action ever runs — --help, --version, a parse error — no longer prefetch CLI update metadata. Nothing displays an update notification on those paths, so I read it as dead work removed rather than a regression. It does change what those invocations do to the update cache, which is your call rather than mine. Say so and I will restore a bounded fallback.

Tests

Four tests in NuGetPackagePrefetcherTests, each written against main first:

• InFlightPrefetchingCompletesBeforeTheServiceStops — ExecuteTask is still running while a prefetch is in flight, and both callbacks have unwound by the time StopAsync returns. The regression test for the leak; fails deterministically on main.
• ReadOnlyCommandsDisablePackageMetadataPrefetching (ls, ps) — resolves the real commands from the real DI graph. Fails deterministically on main.
• ReadOnlyCommandsStartNoPrefetching (ls, ps) — drives the real prefetcher with those instances. Only deterministic once the prefetch tasks are awaited, since that is what makes ExecuteTask a barrier; on main it fails for ps but can pass for ls on scheduling luck. Its value is as a guard after the fix.
• CommandSelectedAfterABannerLengthDelayStillDisablesPrefetching — waits 1500 ms before selecting the command. The delay is the point: it has to outlast the timeout that used to be there.

macOS arm64, .NET SDK 10.0.302. Prefetcher plus LsCommandTests and PsCommandTests: 69/69, no warnings. Full Aspire.Cli.Tests (--filter-not-trait quarantined=true --filter-not-trait outerloop=true): 4695 tests, 4661 pass, 32 skipped, 2 failures — CliPathHelperTests.ResolveSymlinkToFullPath_NonLink_ReturnsNormalizedFullPath and InstallationDiscoveryDiscoverAllTests.DiscoverAllAsync_RunningCliIsAlwaysFirst, both of which fail identically on an unmodified checkout here (macOS /private/tmp symlink normalization, and a running-CLI assumption).

I did not touch stdin or the TTY. #16791 / #17562 is the complementary fix for why an orphaned helper ends up stopped rather than exiting; this stops it being orphaned.

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
  • No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
  • No
• Did you add public API?
  • Yes
  • No
• Does the change make any security assumptions or guarantees?
  • Yes
  • No

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 18958

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 18958"

[Rafał Araszkiewicz (Arasz)]

@microsoft-github-policy-service agree

[Copilot]

Pull request overview

Prevents NuGet prefetch helpers from outliving Aspire CLI invocations.

Changes:

• Tracks and awaits package prefetch tasks during shutdown.
• Disables metadata prefetching for ls and ps.
• Adds lifecycle and command-specific regression tests.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
src/Aspire.Cli/NuGet/NuGetPackagePrefetcher.cs	Tracks prefetch task completion.
src/Aspire.Cli/Commands/LsCommand.cs	Disables metadata prefetching.
src/Aspire.Cli/Commands/PsCommand.cs	Disables metadata prefetching.
tests/Aspire.Cli.Tests/NuGet/NuGetPackagePrefetcherTests.cs	Adds regression coverage.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/Aspire.Cli/NuGet/NuGetPackagePrefetcher.cs:91

• Awaiting these tasks does not guarantee that the CLI waits for cleanup. Release builds give the entire host shutdown only 200 ms (Program.cs:317-323), and BackgroundService.StopAsync stops waiting for ExecuteTask when that token expires. If that budget is exhausted before this service runs, or before its cancellation continuation kills the helper, Program.cs:1249 can still return with the prefetch running—the orphaning path this method claims to prevent. The new test uses CancellationToken.None, so it cannot catch this. Please make helper termination an owned shutdown step that cannot be abandoned by the host timeout, while bounding any post-kill drain separately.

    private static async Task PreventOrphanedPrefetchingAsync(List<Task> prefetchTasks, CancellationToken stoppingToken)
    {
        try
        {
            await Task.WhenAll(prefetchTasks);

src/Aspire.Cli/Commands/LsCommand.cs:23

• aspire ls does not list resources from a running AppHost; it discovers candidate AppHost project files in the workspace (as SharedCommandStrings.LsCommandDescription and ExecuteAsync describe). Correct this rationale so it does not imply that ls depends on a running application.

    /// <summary>
    /// LsCommand lists resources of a running AppHost and never uses template package metadata.
    /// </summary>

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[Rafał Araszkiewicz (Arasz)]

Both suppressed comments were worth answering.

LsCommand.cs:23 — wrong, fixed in 8122513. LsCommandDescription reads "List candidate AppHost project files in the workspace (use aspire ps for running resources)", and ExecuteAsync calls FindAppHostProjectsAsync over an AppHostDiscoveryScope. I had described ps, not ls. The PsCommand doc stays as-is — PsCommandStrings.Description is literally "List running AppHosts".

NuGetPackagePrefetcher.cs:91 — accurate, and deliberately not fixed here. The 200ms cap (Program.cs:317-323) can expire before the kill lands, so this is best-effort teardown rather than a guarantee, and the PR description says so rather than claiming otherwise. I have not made termination un-abandonable because that cap is a documented decision — "to ensure the CLI exits quickly" — and an un-abandonable step would let a slow child teardown hold the CLI open past it. Today the helper is abandoned on every invocation that starts one; after this change, only if teardown overruns 200ms.

If you want the hard guarantee, I will add it as an explicit shutdown step with its own budget. Your call on the tradeoff.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[James Newton-King (JamesNK)]

Updated the delayed command-selection regression test to use FakeTimeProvider, so it no longer requires a real Task.Delay.

Further improvements to make NuGet metadata prefetching opt-in per command are tracked in #18965.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

tests/Aspire.Cli.Tests/NuGet/NuGetPackagePrefetcherTests.cs:307

• This does not reproduce the timeout being regressed. The removed implementation used a real CancellationTokenSource(TimeSpan.FromSeconds(1)), so advancing this fake clock consumes no wall time; against that implementation the command is selected immediately and the test still passes. Wait longer than the old real timeout and assert ExecuteTask is still pending before selecting the command, so the test also proves the service did not silently stop waiting.

        timeProvider.Advance(TimeSpan.FromMilliseconds(1500));

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[github-actions]

🔍 CI Failure Analysis: Transient Infrastructure Failure

The CI build failed due to transient infrastructure issues.

Failed jobs:

• Tests / Hosting.SqlServer / Hosting.SqlServer (windows-latest) — The .NET SDK 10.0.201 download from builds.dotnet.microsoft.com failed with 'Error while copying content to a stream. Unable to read data from the transport connection: The connection was closed.' All fallback URLs returned 404. This is a transient network connectivity failure unrelated to PR changes. (transient-infra)

If a rerun was not already requested automatically, visit the workflow run page to rerun the failed jobs manually.