fix(e2e): move Aspire restore proof to fixed CLI build

[rickylabs]

Summary

Root-cause the intermittent Aspire restore cancellation and move the published-canary workflow from affected Aspire CLI 13.4.6 to the exact tested daily build containing the upstream NuGet helper lifecycle repair. PR #1305 S1 log capture is reused; its cache and retry mitigation is superseded by its own failing controls.

Scope

• Archetype / area: CLI E2E and release workflow
• Closes fix(e2e): runtime.aspire-restore intermittently hangs 15min to timeout against NuGet feeds — 3 occurrences in one day, local and cloud #1227
• Supersedes the mitigation in fix(e2e): eliminate intermittent Aspire restore cancellation #1305 after the owner closes it

Slices

• S1 Capture and read the failing Aspire CLI logs — 6c1cd60
• S2 Name cancellation ownership, establish upstream and daily state, and pin the fixed CLI — 6c1cd60, 94b23bf
• S3 Prove five consecutive published-canary workflows green — 5c9a656

Validation

• Focused Aspire workflow policy test — 2 passed, 0 failed
• Targeted Quickstart walk test — 5 passed, 0 failed
• Scoped TypeScript check, lint, and formatting — 0 findings
• Harness Markdown formatting — 12 files, 0 findings
• Exact official installer dry-run — versioned archive and checksum path resolved
• Isolated 13.4.6 SDK restore through fixed daily CLI — exit 0 in 13.06 seconds; no new helper leaked
• Five consecutive full published-canary workflows — green at immutable head 5c9a6567a08432afdaa79dd313226fd55842e1f4; restore/start passed in 18.596–21.200 seconds

Proof runs: 30966854400, 30967458694, 30967946498, 30968431907, 30968898863.

Harness

• Run dir: .llm/runs/research-aspire-restore-root-cause--1227/
• Phase: impl complete; D6 owner waiver means no local PLAN-EVAL
• Route: openai / gpt-5.6-sol / xhigh
• Structured implementation evidence: IMPL comment

Acceptance evidence

issue: 1227
entries:
  - box: "A hung feed no longer consumes 30 minutes before failing the gate."
    evidence: "Five consecutive full workflows at immutable SHA 5c9a6567a completed restore/start in 18.596–21.200s: https://github.com/rickylabs/netscript/pull/1308#issuecomment-5186835227"
  - box: "The pinned Aspire.Hosting packages restore from cache when the network feed stalls (or an equivalent mitigation, decided and documented)."
    evidence: "Equivalent causal fix: exact daily 13.5.0-preview.1.26404.10 contains microsoft/aspire#18958 and passed all five no-retry published-canary walks: https://github.com/rickylabs/netscript/pull/1308#issuecomment-5186835227"
  - box: "Restore-infra failures are distinguishable from product failures in the suite summary."
    evidence: "The bounded quickstart.4-aspire-restore-start verdict and retained Aspire CLI log identify NetScript's deadline cancellation separately from the hung Aspire NuGet helper: https://github.com/rickylabs/netscript/pull/1308#issuecomment-5186835227"

Drift / Debt

• Exact daily preview pin is temporary operational debt until a stable Aspire build contains Stop leaking orphaned aspire-managed NuGet search helpers microsoft/aspire#18958.
• Existing user-owned deno.lock churn is excluded.

Definition of Done

• A failing Aspire CLI log is captured and read with the blocked component and cancellation initiator named
• Existing upstream defect and merged fix are linked; no duplicate issue is needed
• Stable and daily currency is established, including commit ancestry
• The implementation follows the diagnosis rather than retrying or prewarming the symptom
• Five consecutive full published-canary workflows pass at one immutable head

[rickylabs]

[PHASE: RESEARCH]

Root cause established: NetScript terminates Aspire at the 180-second gate, but the blocked operation is Aspire 13.4.6 BundleNuGetService awaiting bundled NuGet restore after earlier stopped metadata-search helpers poison NuGet locks.

Findings

1. Failing run 30962130571 logs Termination signal received before TaskCanceledException in BundleNuGetService.RestorePackagesAsync; NetScript aspire-walk.ts owns the abort deadline.
2. aspire ls leaks orphaned aspire-managed nuget search helpers (macOS) that accumulate over days and deadlock unrelated dotnet restore via NuGet lock contention microsoft/aspire#18948 matches exact 13.4.6, WSL, stopped aspire-managed search helpers, and restore deadlock. Stop leaking orphaned aspire-managed NuGet search helpers microsoft/aspire#18958 is the merged repair.
3. PR fix(e2e): eliminate intermittent Aspire restore cancellation #1305 prewarm/retry is falsified by runs 30964226683 and 30965320792; the latter failed both preseeded attempts at exactly 180 seconds.
4. Exact daily 13.5.0-preview.1.26404.10 contains the repair and restored the unchanged 13.4.6 SDK graph in 13.06 seconds without leaking a helper.

Next

• Prove the exact pin with five consecutive full published-canary workflows at head 94b23bf.

[rickylabs]

[PHASE: IMPL] [VERDICT: COMPLETE]

Root cause is established and the diagnosis-derived version move passed the owner-required repeated published-canary proof.

Findings

1. Cancellation ownership — Aspire CLI 13.4.6 stalls in bundled aspire-managed nuget restore / BundleNuGetService.RestorePackagesAsync. NetScript's outer deadline sends SIGTERM; Aspire's ConsoleCancellationManager cancels the shared token, kills the child tree, and surfaces TaskCanceledException / exit 6.
2. Underlying defect — aspire ls leaks orphaned aspire-managed nuget search helpers (macOS) that accumulate over days and deadlock unrelated dotnet restore via NuGet lock contention microsoft/aspire#18948 and merged #18958 identify the affected 13.4.6 orphaned NuGet metadata-helper lifecycle and lock contention. Open #18965 tracks narrower command-scoped prefetching.
3. PR fix(e2e): eliminate intermittent Aspire restore cancellation #1305 control — run 30965320792 falsified cache + retry as a fix: both preseeded 180-second attempts hung in the same bundled restore and exited 6. Its S1 log capture and independent service-scoped response probe are reused; its cache/retry diagnosis is superseded.
4. Version move — published E2E pins exact daily 13.5.0-preview.1.26404.10, which contains #18958, while retaining the 13.4.6 AppHost SDK/package graph. Replace this preview pin with fixed stable 13.5 when released.

Five consecutive greens

All five runs used immutable head 5c9a6567a08432afdaa79dd313226fd55842e1f4; exact CLI preflight, full scaffold runtime, and the seven-verdict Quickstart walk passed in every run.

• https://github.com/rickylabs/netscript/actions/runs/30966854400
• https://github.com/rickylabs/netscript/actions/runs/30967458694
• https://github.com/rickylabs/netscript/actions/runs/30967946498
• https://github.com/rickylabs/netscript/actions/runs/30968431907
• https://github.com/rickylabs/netscript/actions/runs/30968898863

Handoff

• PR fix(e2e): move Aspire restore proof to fixed CLI build #1308 is ready for external IMPL-EVAL; it remains draft/status:impl until that separate-session gate.
• Close fix(e2e): eliminate intermittent Aspire restore cancellation #1305 as superseded after owner review.
• The exact-signature upstream issue draft is retained at .llm/runs/research-aspire-restore-root-cause--1227/upstream-issue-draft.md; do not file without owner approval.

[rickylabs]

[PHASE: IMPL]

Root cause is established, the affected Aspire build is replaced by the exact daily containing the upstream repair, and five consecutive published-canary workflows are green at one immutable head.

Diagnosis

• The visible TaskCanceledException is secondary. In the failing CLI log, NetScript's 180-second gate first sends the termination signal (Termination signal received, requesting cancellation); Aspire then cancels Process.WaitForExitAsync while BundleNuGetService.RestorePackagesAsync is waiting for the bundled aspire-managed nuget restore helper.
• The operation that hangs is Aspire 13.4.6's NuGet helper lifecycle. Its discarded metadata-prefetch tasks can leave stopped/orphaned aspire-managed nuget search helpers holding or starving NuGet scratch/global locks. This exact signature is tracked by microsoft/aspire#18948.
• microsoft/aspire#18958, merged as 7921e8689f5f6cec0fa4f815f9c4a04f7b875264, tracks, cancels, awaits, and reaps those helpers. Stable 13.4.6 predates that repair. The pinned daily 13.5.0-preview.1.26404.10 is 25 commits ahead of the repair merge.
• PR fix(e2e): eliminate intermittent Aspire restore cancellation #1305's S1 CLI-log retention is reused. Its package-cache preseed and signature retry are superseded: run 30964226683 hung later in nested aspire start despite a verified cache hit, and run 30965320792 hung both restore attempts after verified preseed.

Slices landed

1. Captured and read the failing cloud and local Aspire CLI logs; named both the blocked component and cancellation initiator.
2. Established stable/daily currency and moved the production walk to the exact fixed daily build, with exact-version preflight, aspire doctor, and CLI-log retention.
3. Carried PR fix(e2e): eliminate intermittent Aspire restore cancellation #1305's unrelated Quickstart service-response correction so the full seven-step walk measures the service endpoint rather than a database aggregate.
4. Proved five consecutive full 0.0.5-canary.10 workflows green at immutable head 5c9a6567a08432afdaa79dd313226fd55842e1f4:

Run	Workflow	Quickstart restore/start
1	30966854400 — green	19.693s
2	30967458694 — green	20.166s
3	30967946498 — green	20.271s
4	30968431907 — green	21.200s
5	30968898863 — green	18.596s

Every run reports Aspire CLI 13.5.0-preview.1.26404.10, scaffold.runtime at passed=71 failed=0, and the Quickstart walk at passed=9 failed=0. There is no retry path in this branch.

Validation

• Focused workflow policy test: 2 passed, 0 failed.
• Targeted Quickstart walk test: 5 passed, 0 failed.
• Scoped TypeScript check, lint, and format: 0 findings.
• Official exact-version installer dry-run: archive and checksum resolved.
• Isolated empty-cache restore using the fixed daily against the 13.4.6 SDK/package train: exit 0 in 13.06s; no new helper leaked.
• Five consecutive published-canary proof workflows: green, same SHA, with CLI logs retained as artifacts.

Next

• External, separate-session IMPL-EVAL remains required before status:ready-merge.
• Treat the preview pin as temporary operational debt; move back to stable when a stable Aspire release includes Stop leaking orphaned aspire-managed NuGet search helpers microsoft/aspire#18958.
• PR fix(e2e): move Aspire restore proof to fixed CLI build #1308 supersedes PR fix(e2e): eliminate intermittent Aspire restore cancellation #1305's mitigation. PR fix(e2e): eliminate intermittent Aspire restore cancellation #1305 can be closed by the owner; its S1 log-retention work remains incorporated here.