Replace template harness with CLI E2E coverage by davidfowl · Pull Request #16305 · microsoft/aspire

davidfowl · 2026-04-18T23:33:04Z

Description

This removes the standalone tests/Aspire.Templates.Tests harness and moves the template coverage we still want to keep into Aspire CLI E2E tests.

The old harness provided broad matrix coverage, but it was hard to debug, had dedicated CI/Helix plumbing, and duplicated scenarios that are better validated through the real CLI. This PR keeps high-signal behavioral coverage, makes failures easier to diagnose, and intentionally drops low-signal matrix breadth.

Test coverage changes

Old test coverage	New test coverage	Change
`BuildAndRunTemplateTests.BuildAndRunAspireTemplate`	`SmokeTests.CreateAndRunDotNetEmptyTemplateProject`, `SmokeTests.CreateAndRunDotNetAppHostTemplateProject`	Preserved through real `aspire new` / `aspire start` CLI E2E coverage.
`BuildAndRunTemplateTests.BuildAndRunAspireTemplateWithCentralPackageManagement`	`DotNetTemplateProjectFileBehaviorTests.DotNetTemplateWithCentralPackageManagementBuildsAndStarts`	Preserved.
`BuildAndRunTemplateTests.BuildAndRunAspireTemplateWithExplicitSdkReference`	`DotNetTemplateProjectFileBehaviorTests.DotNetTemplateWithExplicitSdkReferenceBuildsAndStarts`	Preserved for both explicit SDK-reference modes.
`BuildAndRunTemplateTests.CreateAndModifyAspireAppHostTemplate`	`DotNetTemplateProjectFileBehaviorTests.DotNetAppHostTemplateBuildsWithAppHostPackageVersionOverride`	Preserved for representative package version override cases.
`BuildAndRunTemplateTests.ProjectWithNoHTTPSRequiresExplicitOverrideWithEnvironmentVariable`	`DotNetTemplateTransportTests.NoHttpsTemplateRequiresAllowUnsecuredTransport`	Preserved and made more robust by asserting captured detach logs instead of relying on exact terminal timing.
`AppHostTemplateTests.EnsureProjectsReferencing8_1_0AppHostWithNewerWorkloadCanBuild`	`DotNetTemplateProjectFileBehaviorTests.DotNetAppHostTemplateBuildsWithAppHostPackageVersionOverride("8.1.0")`	Preserved.
`LocalhostTldHostnameTests.LocalhostTld_GeneratesDnsCompliantHostnames`	`DotNetTemplateLocalhostTldTests.LocalhostTldGeneratesDnsCompliantHostnames`	Preserved, including the restored edge-case hostnames.
No equivalent debuggable CLI E2E smoke for `aspire-apphost-singlefile`	`DotNetSingleFileAppHostTemplateTests.SingleFileAppHostTemplateBuildsAndStarts`	Improved: adds create/build/start coverage for the single-file AppHost template.
`BuildAndRunStarterTemplateBuiltInTest.BuildAndRunStarterTemplateBuiltInTest_Test`	`StarterTemplateBehaviorTests.StarterTemplateCanGenerateAndRunBuiltInTests`	Preserved for representative frameworks: xUnit v2, xUnit v3mtp, NUnit, MSTest.
`StarterTemplateRunTestsBase.ResourcesShowUpOnDashboard`, `WebFrontendWorks`, `ApiServiceWorks`	`SmokeTests.CreateAndRunDefaultAspireStarterProject`, `SmokeTests.StarterJsonContractsAndEndpointsRespond`	Preserved and improved with `aspire start`, `aspire ps`, and `aspire describe` JSON contract assertions plus HTTP 200 endpoint validation.
`StarterTemplateWithRedisCacheTests` and previous-TFM Redis variants	`StarterTemplateBehaviorTests.StarterTemplateWithRedisHasCacheResourceAndApiResponds`	Preserved for current representative coverage; previous-TFM duplication dropped.
`NewUpAndBuildSupportProjectTemplatesTests` for `aspire-xunit`, `aspire-nunit`, `aspire-mstest`	`SupportProjectTemplateBehaviorTests.SupportProjectTemplatesRunAgainstGeneratedAppHost`	Improved: generated support projects now run generated integration tests with `dotnet test`, not only build.
xUnit support-template variants including v2, v3, v3mtp	`SupportProjectTemplateBehaviorTests.SupportProjectTemplatesRunAgainstGeneratedAppHost` for default xUnit and xUnit v3	Reduced intentionally; keeps representative xUnit support-template execution coverage.
`PerTestFrameworkTemplatesTests` and broad starter project-name matrix	Representative starter/support framework cases in `StarterTemplateBehaviorTests` and `SupportProjectTemplateBehaviorTests`	Reduced intentionally; dropped broad project-name matrix in favor of debuggable behavior coverage.
`NewUpAndBuildStandaloneTemplateTests` broad `new + build` matrix	`SmokeTests`, `DotNetTemplateBehaviorTests`, and `TemplateVariantSmokeTests`	Reduced intentionally; replaced broad build-only matrix with targeted runtime smoke coverage.
One-off CLI smoke classes for bundle, empty AppHost, Java, JS React, Python React, TypeScript empty, TypeScript starter	`SmokeTests` and `TemplateVariantSmokeTests`	Consolidated into shardable smoke classes.
Python/React and TypeScript template smoke	`TemplateVariantSmokeTests.CreateAndRunPythonReactProject`, `CreateAndRunTypeScriptEmptyAppHostProject`, `CreateAndRunTypeScriptStarterProject`	Preserved and improved with `npm run build` before start.
Java empty AppHost and JS React smoke	`TemplateVariantSmokeTests.CreateAndRunJavaEmptyAppHostProject`, `CreateAndRunJsReactProject`	Preserved.
No direct replacement for previous-TFM starter/Redis duplication	None	Dropped intentionally. Current-TFM behavior remains covered.
No direct replacement for the old `TestSdk` workload/SDK matrix	None	Dropped intentionally. This PR keeps behavior-focused CLI E2E coverage instead of broad workload matrix coverage.

Other changes

Deleted tests/Aspire.Templates.Tests and its helper/fixture infrastructure.
Removed the dedicated template-test CI/Helix path, including the old templatestests category.
Removed stale custom SDK/archive setup used only by the deleted harness.
Updated CI docs and tests/README.md to point at the CLI E2E replacement coverage.
Replaced the old internal-build template smoke with signed CLI archive verification in eng/scripts/verify-cli-archive.ps1.
Hardened shared CLI E2E Dockerfiles by making apt package index fetches retry and fail fast on update errors.

Debuggability improvements

The replacement tests use normal CLI E2E infrastructure:

[CaptureWorkspaceOnFailure]
preserved generated project workspaces
terminal recordings
captured JSON/log files
shared AspireStartAsync / AspireStopAsync helpers
real CLI command flow instead of a separate template harness

Validation

./restore.sh
dotnet build tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj --nologo --verbosity:minimal
targeted CLI E2E runs for the changed template behavior tests
docker build --target runtime --build-arg SKIP_SOURCE_BUILD=true -f tests/Shared/Docker/Dockerfile.e2e -t aspire-e2e-dockerfile-check .

Fixes # (issue)

Checklist

Is this feature complete?
- Yes. Ready to ship.
- No. Follow-up changes expected.
Are you including unit tests for the changes and scenario tests if relevant?
- Yes
- No
Did you add public API?
- Yes
  - If yes, did you have an API Review for it?
    - Yes
    - No
  - Did you add <remarks /> and <code /> elements on your triple slash comments?
    - Yes
    - No
- No
Does the change make any security assumptions or guarantees?
- Yes
  - If yes, have you done a threat model and had a security review?
    - Yes
    - No
- No
Does the change require an update in our Aspire docs?
- Yes
  - Link to aspire.dev issue:
    - New issue
- No

github-actions · 2026-04-18T23:33:13Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 16305

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 16305"

github-actions · 2026-04-18T23:57:19Z

Re-running the failed jobs in the CI workflow for this pull request because 2 jobs were identified as retry-safe transient failures in the CI run attempt.
GitHub was asked to rerun all failed jobs for that attempt, and the rerun is being tracked in the rerun attempt.
The job links below point to the failed attempt jobs that matched the retry-safe transient failure rules.

Tests / Hosting.Docker / Hosting.Docker (windows-latest) - Failed step 'Checkout code' matched the transient annotation allowlist.
Tests / StackExchange.Redis.DistributedCaching / StackExchange.Redis.DistributedCaching (windows-latest) - Failed step 'Set up job' matched the transient annotation allowlist.

radical

Review: 5 findings (1 coverage gap high, 2 coverage gap medium, 1 flaky pattern medium, 1 correctness low). Inline comments on 4 findings above.

Additional finding (file-level):

The deleted EmptyAppHostTemplateTests.cs, JavaEmptyAppHostTemplateTests.cs, and TypeScriptEmptyAppHostTemplateTests.cs ran aspire new aspire-apphost → build → run as a happy-path smoke test. TemplateVariantSmokeTests covers JS/Python/TS starters, but the .NET aspire-apphost empty template’s build+run path has no replacement coverage. The only related test (NoHttpsTemplateRequiresAllowUnsecuredTransport) is a negative test with --no-https. Consider adding a happy-path smoke for the .NET empty apphost template.

radical · 2026-04-22T19:46:18Z

We run some of the template tests in the internal build -

aspire/eng/pipelines/templates/BuildAndTest.yml

Lines 201 to 217 in 69e9ebe

    
           - script: ${{ parameters.buildScript }} 
        
                     -build 
        
                     -restore 
        
                     -test 
        
                     -configuration ${{ parameters.buildConfig }} 
        
                     /bl:${{ parameters.repoLogPath }}/BuildTemplatesTests.binlog 
        
                     $(_OfficialBuildIdArgs) 
        
                     $(_InternalBuildArgs) 
        
                     /p:SkipTests=false 
        
                     -projects $(Build.SourcesDirectory)\tests\Aspire.Templates.Tests\Aspire.Templates.Tests.csproj 
        
             env: 
        
               RunOnlyBasicBuildTemplateTests: true 
        
               # test root path for template test projects 
        
               DEV_TEMP: $(Build.SourcesDirectory)\.. 
        
               DOTNET_ROOT: $(Build.SourcesDirectory)\.dotnet 
        
               TEST_LOG_PATH: $(Build.SourcesDirectory)\artifacts\log\$(_BuildConfig)\Aspire.Templates.Tests 
        
             displayName: 🟣Run Template tests

. We'll need a replacement for that.

davidfowl · 2026-04-22T23:36:31Z

We run some of the template tests in the internal build -

Why?

radical

Review feedback from walkthrough.

radical

Review comments for removing remaining Aspire.Templates.Tests/custom test SDK infrastructure and replacing the removed AzDO template coverage with direct CLI E2E coverage.

github-actions · 2026-05-01T06:06:48Z

Re-running the failed jobs in the CI workflow for this pull request because 4 jobs were identified as retry-safe transient failures in the CI run attempt.
GitHub was asked to rerun all failed jobs for that attempt, and the rerun is being tracked in the rerun attempt.
The job links below point to the failed attempt jobs that matched the retry-safe transient failure rules.

Tests / Cli.EndToEnd-KubernetesDeployWithSqlServerTests / Cli.EndToEnd-KubernetesDeployWithSqlServerTests (ubuntu-latest) - Failed step 'Run nuget dependent tests (Linux/macOS)' will be retried because the job log matched a configurable test-retry pattern: MCR registry rate limiting (HTTP 403)
Tests / Cli.EndToEnd-StarterTemplateBehaviorTests / Cli.EndToEnd-StarterTemplateBehaviorTests (ubuntu-latest) - Test execution failure will be retried because 1 failed test(s) matched transient test failure patterns (MCR registry rate limiting (HTTP 403)).
Tests / Cli.EndToEnd-DotNetTemplateLocalhostTldTests / Cli.EndToEnd-DotNetTemplateLocalhostTldTests (ubuntu-latest) - Test execution failure will be retried because 1 failed test(s) matched transient test failure patterns (MCR registry rate limiting (HTTP 403)).
Tests / Cli.EndToEnd-StopNonInteractiveTests / Cli.EndToEnd-StopNonInteractiveTests (ubuntu-latest) - Test execution failure will be retried because 1 failed test(s) matched transient test failure patterns (MCR registry rate limiting (HTTP 403)).

Matched test failure patterns (1 test)

Aspire.Cli.EndToEnd.Tests.KubernetesDeployWithSqlServerTests.DeployK8sWithSqlServer — MCR registry rate limiting (HTTP 403)

github-actions · 2026-05-02T16:30:43Z

Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
GitHub was asked to rerun all failed jobs for that attempt, and the rerun is being tracked in the rerun attempt.
The job links below point to the failed attempt jobs that matched the retry-safe transient failure rules.

Tests / Hosting-1 / Hosting-1 (windows-latest) - Job-level runner or infrastructure failure matched the transient allowlist.

Copilot

Pull request overview

Replaces the dedicated Aspire.Templates.Tests harness with CLI E2E coverage, and rewires CI/Helix/docs to match the new testing model.

Changes:

Deletes the standalone template test project plus its SDK-install and Helix plumbing.
Adds new CLI E2E smoke/behavior tests for starter, AppHost, TypeScript, Java, Python, and .NET template scenarios.
Updates GitHub Actions, AzDO, archive verification, and docs to use the new CLI-driven coverage path.

Findings:

tests/Aspire.Cli.EndToEnd.Tests/Helpers/CliE2ETestHelpers.cs returns the first localhost URL from raw JSON text, which will usually be dashboardUrl, so StarterJsonContractsAndEndpointsRespond validates the dashboard instead of the webfrontend endpoint.
tests/Aspire.Cli.EndToEnd.Tests/StarterTemplateBehaviorTests.cs drops the old Redis starter frontend/cache assertions; the replacement only checks resource presence plus API output.
tests/Aspire.Cli.EndToEnd.Tests/StarterTemplateBehaviorTests.cs no longer covers special-character/unicode project names that the deleted harness used to validate template name sanitization and encoding behavior.
tests/Aspire.Cli.EndToEnd.Tests/DotNetTemplateBehaviorTests.cs removes the old net8/net10 and unsupported-SDK/TFM matrix coverage, leaving no replacement regression coverage for template compatibility across supported TFMs.

Show a summary per file

File	Description
`tests/workloads.proj`	Removed obsolete template-SDK installer wrapper.
`tests/Shared/TemplatesTesting/EnvironmentVariables.cs`	Removed old template SDK env var.
`tests/Shared/TemplatesTesting/BuildEnvironment.cs`	Simplified template test environment/bootstrap logic.
`tests/Shared/InstallSdk.targets`	Removed standalone SDK provisioning target.
`tests/Shared/InstallSdk.props`	Removed SDK install helper props.
`tests/Shared/Hex1bAutomatorTestHelpers.cs`	Tweaked agent prompt handling; added TS restore/typecheck helper.
`tests/Shared/Docker/Dockerfile.e2e-polyglot-java`	Hardened apt update invocation.
`tests/Shared/Docker/Dockerfile.e2e-polyglot-base`	Hardened apt update invocation.
`tests/Shared/Docker/Dockerfile.e2e-podman`	Hardened apt update invocation.
`tests/Shared/Docker/Dockerfile.e2e`	Hardened apt update invocation.
`tests/Shared/Aspire.Templates.Testing.targets`	Removed template-test MSBuild plumbing.
`tests/Shared/Aspire.Templates.Testing.props`	Removed template-test props import.
`tests/README.md`	Updated test infra/local-run documentation.
`tests/Infrastructure.Tests/WorkflowScripts/AutoRerunTransientCiFailuresTests.cs`	Updated workflow fixture expectations.
`tests/Infrastructure.Tests/Shared/TestDataBuilder.cs`	Removed `requiresTestSdk` metadata support.
`tests/Infrastructure.Tests/PowerShellScripts/BuildTestMatrixTests.cs`	Removed `requiresTestSdk` test case.
`tests/helix/send-to-helix-templatestests.targets`	Removed template-tests Helix category.
`tests/helix/send-to-helix-inner.proj`	Removed template-SDK payload/env handling.
`tests/helix/send-to-helix-endtoendtests.targets`	Dropped template-SDK dependency flag.
`tests/helix/send-to-helix-ci.proj`	Removed template-tests category from Helix dispatch.
`tests/helix/send-to-helix-buildonhelixtests.targets`	Removed template-SDK PATH injection.
`tests/helix/send-to-helix-basictests.targets`	Removed template-SDK dependency flag.
`tests/Directory.Build.targets`	Removed template testing targets import.
`tests/Directory.Build.props`	Removed template-test archive/props settings.
`tests/Aspire.Templates.Tests/xunit.runner.json`	Deleted old template test runner config.
`tests/Aspire.Templates.Tests/TestSdk.cs`	Deleted old template SDK enum.
`tests/Aspire.Templates.Tests/TemplateTestsBase.cs`	Deleted old template harness base.
`tests/Aspire.Templates.Tests/TemplateAppFixture.cs`	Deleted old template app fixture.
`tests/Aspire.Templates.Tests/StarterTemplateWithRedisCacheTests.cs`	Deleted old Redis starter run test.
`tests/Aspire.Templates.Tests/StarterTemplateWithRedisCacheTests_PreviousTFM.cs`	Deleted old previous-TFM Redis starter test.
`tests/Aspire.Templates.Tests/StarterTemplateWithRedisCacheFixture.cs`	Deleted old Redis starter fixture.
`tests/Aspire.Templates.Tests/StarterTemplateWithRedisCacheFixture_PreviousTFM.cs`	Deleted old previous-TFM Redis starter fixture.
`tests/Aspire.Templates.Tests/StarterTemplateRunTestsBase.cs`	Deleted old starter run assertions base.
`tests/Aspire.Templates.Tests/StarterTemplateRunTests.cs`	Deleted old starter run test.
`tests/Aspire.Templates.Tests/StarterTemplateRunTests_PreviousTFM.cs`	Deleted old previous-TFM starter run test.
`tests/Aspire.Templates.Tests/StarterTemplateProjectNamesTests.cs`	Deleted old project-name coverage.
`tests/Aspire.Templates.Tests/StarterTemplateFixture.cs`	Deleted old starter fixture.
`tests/Aspire.Templates.Tests/StarterTemplateFixture_PreviousTFM.cs`	Deleted old previous-TFM starter fixture.
`tests/Aspire.Templates.Tests/README.md`	Removed old template harness documentation.
`tests/Aspire.Templates.Tests/PerTestFrameworkTemplatesTests.cs`	Deleted old support-template tests.
`tests/Aspire.Templates.Tests/NewUpAndBuildSupportProjectTemplatesTests.cs`	Deleted old support-template SDK/TFM matrix tests.
`tests/Aspire.Templates.Tests/NewUpAndBuildStandaloneTemplateTests.cs`	Deleted old standalone template SDK/TFM matrix tests.
`tests/Aspire.Templates.Tests/LocalhostTldHostnameTests.cs`	Deleted old localhost TLD tests.
`tests/Aspire.Templates.Tests/EmptyTemplateRunTests.cs`	Deleted old empty-template run test.
`tests/Aspire.Templates.Tests/EmptyTemplateRunFixture.cs`	Deleted old empty-template fixture.
`tests/Aspire.Templates.Tests/Directory.Build.props`	Removed old template-project marker props.
`tests/Aspire.Templates.Tests/BuildAndRunTemplateTests.cs`	Deleted old build/run template tests.
`tests/Aspire.Templates.Tests/BuildAndRunStarterTemplateBuiltInTest.cs`	Deleted old built-in starter test coverage.
`tests/Aspire.Templates.Tests/Aspire.Templates.Tests.csproj`	Deleted old template test project.
`tests/Aspire.Templates.Tests/AppHostTemplateTests.cs`	Deleted old AppHost template compatibility test.
`tests/Aspire.EndToEnd.Tests/IntegrationServicesFixture.cs`	Switched E2E fixture to default build environment.
`tests/Aspire.EndToEnd.Tests/Directory.Build.props`	Removed template-test-specific props.
`tests/Aspire.Cli.Tests/DotNet/ProcessExecutionTests.cs`	Relaxed Windows timeout for exit-wait test.
`tests/Aspire.Cli.EndToEnd.Tests/TypeScriptStarterTemplateTests.cs`	Folded into consolidated smoke coverage.
`tests/Aspire.Cli.EndToEnd.Tests/TypeScriptReusablePackageTests.cs`	Reused new TS restore/typecheck helper.
`tests/Aspire.Cli.EndToEnd.Tests/TypeScriptEmptyAppHostTemplateTests.cs`	Folded into consolidated smoke coverage.
`tests/Aspire.Cli.EndToEnd.Tests/TemplateVariantSmokeTests.cs`	Added consolidated non-default template smoke tests.
`tests/Aspire.Cli.EndToEnd.Tests/StarterTemplateBehaviorTests.cs`	Added starter/support-template behavior coverage.
`tests/Aspire.Cli.EndToEnd.Tests/SmokeTests.cs`	Expanded core smoke coverage and JSON contract checks.
`tests/Aspire.Cli.EndToEnd.Tests/PythonReactTemplateTests.cs`	Folded into consolidated smoke coverage.
`tests/Aspire.Cli.EndToEnd.Tests/JsReactTemplateTests.cs`	Folded into consolidated smoke coverage.
`tests/Aspire.Cli.EndToEnd.Tests/JavaEmptyAppHostTemplateTests.cs`	Folded into consolidated smoke coverage.
`tests/Aspire.Cli.EndToEnd.Tests/Helpers/CliE2ETestHelpers.cs`	Added workspace/JSON assertion helpers.
`tests/Aspire.Cli.EndToEnd.Tests/Helpers/CliE2EAutomatorHelpers.cs`	Added new subcommand/run/capture helpers; changed stop behavior.
`tests/Aspire.Cli.EndToEnd.Tests/EmptyAppHostTemplateTests.cs`	Folded into consolidated smoke coverage.
`tests/Aspire.Cli.EndToEnd.Tests/DotNetTemplateBehaviorTests.cs`	Added .NET template behavior regression coverage.
`tests/Aspire.Cli.EndToEnd.Tests/DescribeCommandTests.cs`	Switched to shared start/stop helpers.
`tests/Aspire.Cli.EndToEnd.Tests/BundleSmokeTests.cs`	Folded into consolidated smoke coverage.
`tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj`	Removed skip gate from CLI E2E project.
`src/Aspire.ProjectTemplates/README.md`	Pointed docs to CLI E2E coverage.
`playground/AspireWithMaui/AspireWithMaui.slnx`	Removed deleted template test project.
`eng/Versions.props`	Replaced template-test SDK version props with Helix runtime SDK prop.
`eng/testing/CITestsProperties.props`	Removed `requiresTestSdk` property registry entry.
`eng/TestEnumerationRunsheetBuilder/TestEnumerationRunsheetBuilder.targets`	Removed special template-test enumeration skip.
`eng/scripts/verify-cli-archive.ps1`	Expanded archive verification to starter + hidden AppHost templates.
`eng/pipelines/templates/BuildAndTest.yml`	Replaced template harness runs with CLI E2E subset and archive staging.
`eng/pipelines/azure-pipelines.yml`	Passed Windows target RIDs into shared pipeline template.
`eng/pipelines/azure-pipelines-unofficial.yml`	Passed Windows target RIDs into shared pipeline template.
`dogfood.sh`	Switched dogfood SDK path to repo-local `.dotnet`.
`docs/ci/TestingOnCI.md`	Updated CI test-matrix docs for CLI E2E model.
`docs/ci/azdo-public-pipeline.md`	Updated AzDO/Helix docs after template harness removal.
`Aspire.slnx`	Removed deleted template test project.
`.github/workflows/tests.yml`	Always enumerates CLI E2E tests; updated skipped-job gating.
`.github/workflows/run-tests.yml`	Removed template SDK install step and old artifact path upload.
`.github/CODEOWNERS`	Removed ownership for deleted template-test paths.
`.github/actions/enumerate-tests/action.yml`	Updated enumerate-tests action docs/examples.

Copilot's findings

Files reviewed: 86/86 changed files
Comments generated: 4

github-actions · 2026-05-03T06:31:07Z

Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
GitHub was asked to rerun all failed jobs for that attempt, and the rerun is being tracked in the rerun attempt.
The job links below point to the failed attempt jobs that matched the retry-safe transient failure rules.

Tests / Hosting.JavaScript / Hosting.JavaScript (windows-latest) - Job-level runner or infrastructure failure matched the transient allowlist.

github-actions · 2026-05-03T16:09:09Z

Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
GitHub was asked to rerun all failed jobs for that attempt, and the rerun is being tracked in the rerun attempt.
The job links below point to the failed attempt jobs that matched the retry-safe transient failure rules.

Tests / Cli.EndToEnd-StartStopTests / Cli.EndToEnd-StartStopTests (ubuntu-latest) - Failed step 'Run nuget dependent tests (Linux/macOS)' will be retried because the job log matched a configurable test-retry pattern: MCR registry rate limiting (HTTP 403)

IEvangelist

Really nice cleanup — replacing the heavyweight template harness with focused CLI E2E coverage is a meaningful simplification and the new test structure reads much more cleanly. The signed-archive verification step is a great addition. Approving with one small finding worth considering; the rest of the substantial feedback already on the PR from @radical and @copilot covers the remaining points well.

Copilot · 2026-05-04T12:34:48Z

Review: 5 findings (1 coverage gap high, 2 coverage gap medium, 1 flaky pattern medium, 1 correctness low). Inline comments on 4 findings above.

Additional finding (file-level):...

The .NET aspire-apphost empty template's build+run path is already covered by CreateAndRunDotNetAppHostTemplateProject in SmokeTests.cs (added in a later commit). It does aspire new aspire-apphost → cd → aspire start → aspire stop, which is the happy-path smoke that was previously covered by EmptyAppHostTemplateTests.cs.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

7k6vsblobprodcus337.vsblob.vsassets.io
- Triggering command: /usr/share/dotnet/dotnet dotnet build tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj --nologo --verbosity quiet (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

radical

Deep-review pass on top of the existing feedback. Skipping items already covered in the prior review threads (template framework matrix narrowing, hostname edge cases, CPM/SDK/version-override coverage gaps, parameters.targetRids plumbing, requiresTestSdk / workloads.proj / InstallSdk.* cleanup, CODEOWNERS dangling entry, webfrontend silent-skip).

A few additional items inline below. Two non-anchorable notes:

Stale dotnet-tests process killer — .github/skills/fix-flaky-test/run-test-repeatedly.sh:135-136 and .github/skills/fix-flaky-test/run-test-repeatedly.ps1:128-130 still hunt for dotnet-tests processes. With the custom test SDK install fully removed by this PR, that's dead code now. Worth cleaning up here while we're touching the surrounding test infra.
Namespace cleanup in tests/Shared/TemplatesTesting/* — those shared helpers still use the Aspire.Templates.Tests namespace, even though the project of that name is gone. They are now consumed by Aspire.EndToEnd.Tests, Aspire.Acquisition.Tests, Aspire.Dashboard.Tests, etc. Suggest renaming the namespace to something neutral like Aspire.Tests.TemplatesTesting so it reflects general-purpose helpers rather than a deleted project.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Agent-Logs-Url: https://github.com/microsoft/aspire/sessions/0d859769-a001-44c7-b373-4e3f348e6614 Co-authored-by: IEvangelist <7679720+IEvangelist@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rewrite Ubuntu archive and security apt sources to the Azure mirror before retrying installs in the shared CLI E2E Docker helper. This avoids repeated CI failures when archive.ubuntu.com and security.ubuntu.com are unreachable from GitHub-hosted runners, while allowing APT_RETRY_UBUNTU_MIRROR to override or disable the rewrite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fail loudly when the signed win-x64 CLI archive is missing, expand AzDO template coverage, and document the win-arm64 staging-only path. Harden Bun installer retries and wait for apiservice readiness before endpoint assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-05-05T06:34:58Z

🎬 CLI E2E Test Recordings — 82 recordings uploaded (commit e5eda56)

View all recordings

Status	Test	Recording
❔	AddPackageInteractiveWhileAppHostRunningDetached	▶️ View Recording
❔	AddPackageWhileAppHostRunningDetached	▶️ View Recording
❔	AgentCommands_AllHelpOutputs_AreCorrect	▶️ View Recording
❔	AgentInitCommand_DefaultSelection_InstallsSkillOnly	▶️ View Recording
❔	AgentInitCommand_MigratesDeprecatedConfig	▶️ View Recording
❔	AspireAddPackageVersionToDirectoryPackagesProps	▶️ View Recording
❔	AspireUpdateRemovesAppHostPackageVersionFromDirectoryPackagesProps	▶️ View Recording
❔	Banner_DisplayedOnFirstRun	▶️ View Recording
❔	Banner_DisplayedWithExplicitFlag	▶️ View Recording
❔	Banner_NotDisplayedWithNoLogoFlag	▶️ View Recording
❔	CertificatesClean_RemovesCertificates	▶️ View Recording
❔	CertificatesTrust_WithNoCert_CreatesAndTrustsCertificate	▶️ View Recording
❔	CertificatesTrust_WithUntrustedCert_TrustsCertificate	▶️ View Recording
❔	ConfigSetGet_CreatesNestedJsonFormat	▶️ View Recording
❔	CreateAndRunAspireStarterProject	▶️ View Recording
❔	CreateAndRunAspireStarterProjectWithBundle	▶️ View Recording
❔	CreateAndRunDefaultAspireStarterProject	▶️ View Recording
❔	CreateAndRunDotNetAppHostTemplateProject	▶️ View Recording
❔	CreateAndRunDotNetEmptyTemplateProject	▶️ View Recording
❔	CreateAndRunEmptyAppHostProject	▶️ View Recording
❔	CreateJavaAppHostWithViteApp	▶️ View Recording
❔	CreateTypeScriptAppHostWithViteApp_UsesConfiguredToolchain	▶️ View Recording
❔	DeployK8sBasicApiService	▶️ View Recording
❔	DeployK8sWithGarnet	▶️ View Recording
❔	DeployK8sWithMySql	▶️ View Recording
❔	DeployK8sWithPostgres	▶️ View Recording
❔	DeployK8sWithRabbitMQ	▶️ View Recording
❔	DeployK8sWithRedis	▶️ View Recording
❔	DeployK8sWithSqlServer	▶️ View Recording
❔	DeployK8sWithValkey	▶️ View Recording
❔	DeployTypeScriptAppToKubernetes	▶️ View Recording
❔	DescribeCommandResolvesReplicaNames	▶️ View Recording
❔	DescribeCommandShowsRunningResources	▶️ View Recording
❔	DetachFormatJsonProducesValidJson	▶️ View Recording
❔	DetachFormatJsonProducesValidJsonWhenRestartingExistingInstance	▶️ View Recording
❔	DoListStepsShowsPipelineSteps	▶️ View Recording
❔	DoctorCommand_DetectsDeprecatedAgentConfig	▶️ View Recording
❔	DoctorCommand_TypeScriptAppHostReportsMissingConfiguredToolchain	▶️ View Recording
❔	DoctorCommand_WithSslCertDir_ShowsTrusted	▶️ View Recording
❔	DoctorCommand_WithoutSslCertDir_ShowsPartiallyTrusted	▶️ View Recording
❔	DotNetAppHostTemplateBuildsWithAppHostPackageVersionOverrideCore	▶️ View Recording
❔	DotNetTemplateCreatesSupportedTargetFrameworksAndOlderSdkRejectsNewerTarget	▶️ View Recording
❔	DotNetTemplateWithCentralPackageManagementBuildsAndStarts	▶️ View Recording
❔	DotNetTemplateWithExplicitSdkReferenceBuildsAndStarts	▶️ View Recording
❔	GlobalMigration_HandlesCommentsAndTrailingCommas	▶️ View Recording
❔	GlobalMigration_HandlesMalformedLegacyJson	▶️ View Recording
❔	GlobalMigration_PreservesAllValueTypes	▶️ View Recording
❔	GlobalMigration_SkipsWhenNewConfigExists	▶️ View Recording
❔	GlobalSettings_MigratedFromLegacyFormat	▶️ View Recording
❔	InitTypeScriptAppHost_AugmentsExistingViteRepoAtRoot	▶️ View Recording
❔	InteractiveCSharpInitCreatesExpectedFiles	▶️ View Recording
❔	InvalidAppHostPathWithComments_IsHealedOnRun	▶️ View Recording
❔	LatestCliCanStartStableChannelAppHost	▶️ View Recording
❔	LatestCliCanStartStableChannelTypeScriptAppHost	▶️ View Recording
❔	LegacySettingsMigration_AdjustsRelativeAppHostPath	▶️ View Recording
❔	LocalhostTldGeneratesDnsCompliantHostnames	▶️ View Recording
❔	LogsCommandShowsResourceLogs	▶️ View Recording
❔	NoHttpsTemplateRequiresAllowUnsecuredTransport	▶️ View Recording
❔	OtelLogsReturnsStructuredLogsFromStarterAppCore	▶️ View Recording
❔	PsCommandListsRunningAppHost	▶️ View Recording
❔	PsFormatJsonOutputsOnlyJsonToStdout	▶️ View Recording
❔	PublishWithConfigureEnvFileUpdatesEnvOutput	▶️ View Recording
❔	PublishWithDockerComposeServiceCallbackSucceeds	▶️ View Recording
❔	PublishWithoutOutputPathUsesAppHostDirectoryDefault	▶️ View Recording
❔	RestoreGeneratesSdkFiles	▶️ View Recording
❔	RestoreGeneratesSdkFiles_WithConfiguredToolchain	▶️ View Recording
❔	RestoreRefreshesGeneratedSdkAfterAddingIntegration	▶️ View Recording
❔	RestoreSupportsConfigOnlyHelperPackageAndCrossPackageTypes	▶️ View Recording
❔	RunFromParentDirectory_UsesExistingConfigNearAppHost	▶️ View Recording
❔	SecretCrudOnDotNetAppHost	▶️ View Recording
❔	SecretCrudOnTypeScriptAppHost	▶️ View Recording
❔	SingleFileAppHostTemplateBuildsAndStarts	▶️ View Recording
❔	StagingChannel_ConfigureAndVerifySettings_ThenSwitchChannels	▶️ View Recording
❔	StartAndWaitForTypeScriptSqlServerAppHostWithNativeAssets	▶️ View Recording
❔	StarterJsonContractsAndEndpointsRespond	▶️ View Recording
❔	StarterTemplateCanGenerateAndRunBuiltInTests	▶️ View Recording
❔	StarterTemplateWithRedisHasCacheResourceAndApiResponds	▶️ View Recording
❔	StopAllAppHostsFromAppHostDirectory	▶️ View Recording
❔	StopNonInteractiveSingleAppHost	▶️ View Recording
❔	StopWithNoRunningAppHostExitsSuccessfully	▶️ View Recording
❔	SupportProjectTemplatesRunAgainstGeneratedAppHost	▶️ View Recording
❔	UnAwaitedChainsCompileWithAutoResolvePromises	▶️ View Recording

_{📹 Recordings uploaded automatically from CI run #25360158323}

davidfowl changed the title ~~Remove remaining CLI template tests~~ Replace template harness with CLI E2E coverage Apr 19, 2026

radical reviewed Apr 20, 2026

View reviewed changes

Comment thread tests/Aspire.Cli.EndToEnd.Tests/DotNetTemplateBehaviorTests.cs Outdated

radical reviewed Apr 20, 2026

View reviewed changes

davidfowl force-pushed the davidfowl/remove-template-tests branch from 526c5ca to 2d6757b Compare April 30, 2026 15:16

github-actions Bot mentioned this pull request Apr 30, 2026

[repo-pulse] 📊 Repo Pulse — Daily Activity Dashboard #16404

Open

radical reviewed May 1, 2026

View reviewed changes

davidfowl force-pushed the davidfowl/remove-template-tests branch from 2d6757b to c282caa Compare May 1, 2026 04:41

davidfowl force-pushed the davidfowl/remove-template-tests branch 2 times, most recently from 0ee22b4 to fb465e6 Compare May 1, 2026 23:43

davidfowl marked this pull request as ready for review May 2, 2026 18:42

davidfowl requested a review from eerhardt as a code owner May 2, 2026 18:42

Copilot AI review requested due to automatic review settings May 2, 2026 18:42

Copilot started reviewing on behalf of davidfowl May 2, 2026 18:42 View session

Copilot AI reviewed May 2, 2026

View reviewed changes

davidfowl force-pushed the davidfowl/remove-template-tests branch from 647b397 to 03aa12d Compare May 3, 2026 05:02

IEvangelist approved these changes May 4, 2026

View reviewed changes

Comment thread tests/Aspire.Cli.EndToEnd.Tests/StarterTemplateBehaviorTests.cs Outdated

Copilot started work on behalf of IEvangelist May 4, 2026 12:29 View session

Copilot finished work on behalf of IEvangelist May 4, 2026 12:36

Copilot AI requested a review from IEvangelist May 4, 2026 12:36

radical reviewed May 4, 2026

View reviewed changes

davidfowl and others added 28 commits May 4, 2026 22:35

Allow empty CI matrix buckets

6a97c38

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Increase Windows timeout for ProcessExecution test

6aaad89

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Apply template test review feedback

b32efe4

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace internal template smoke coverage

8d90aee

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add targeted template E2E coverage

5d98012

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix template E2E rebase fallout

32f6036

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Address template coverage review feedback

4ff5b29

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Update workflow fixture after SDK cleanup

fd5f2ed

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix CLI E2E aspire new prompt handling

6ee87e5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix localhost template path assertion

2188f55

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Resolve generated template output directories

9eb7b66

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use shared start helper in describe E2E tests

639a16a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Harden CLI E2E Docker apt restores

cba5bd7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Address template coverage review feedback

cb04f84

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Restore targeted template TFM coverage

234f56d

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Log CLI install strategy detection in E2E tests

e2b9c8b

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Improve CLI E2E command diagnostics

01166ee

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix CLI E2E agent init prompt handling

9366fc2

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Retry Bun install in CLI E2E Docker images

57bc197

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Restore Aspire E2E build environment behavior

80ad7bf

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace silent skip with Assert.Fail in StarterTemplateBehaviorTests

1e0a670

Agent-Logs-Url: https://github.com/microsoft/aspire/sessions/0d859769-a001-44c7-b373-4e3f348e6614 Co-authored-by: IEvangelist <7679720+IEvangelist@users.noreply.github.com>

Retry apt installs in CLI E2E Docker images

502980d

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Bound apt retry timeouts in CLI E2E Docker images

d912178

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Apply CLI E2E timeout settings to CI metadata

c7165eb

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Increase CLI E2E hang dump timeout

a5c6c93

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Extend CLI E2E timeout budget for cold Docker builds

c57f3bb

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

davidfowl force-pushed the davidfowl/remove-template-tests branch from c7e0e24 to e5eda56 Compare May 5, 2026 05:50

Conversation

davidfowl commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test coverage changes

Other changes

Debuggability improvements

Validation

Checklist

Uh oh!

github-actions Bot commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Apr 18, 2026

Uh oh!

Uh oh!

radical left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

radical commented Apr 22, 2026

Uh oh!

davidfowl commented Apr 22, 2026

Uh oh!

radical left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

radical left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 1, 2026

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 3, 2026

Uh oh!

github-actions Bot commented May 3, 2026

Uh oh!

IEvangelist left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

radical left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davidfowl commented Apr 18, 2026 •

edited

Loading

github-actions Bot commented Apr 18, 2026 •

edited

Loading

Copilot AI commented May 4, 2026 •

edited

Loading