Skip to content

Commit 353aab4

Browse files
Copilotradical
andauthored
[CI] Add new Outerloop workflow for infrequent test runs (#11250)
* Initial plan * Implement CI improvements: rename outerloop to quarantine, add new OuterloopTest attribute and workflow Co-authored-by: radical <1472+radical@users.noreply.github.com> * Update copilot instructions with outerloop tests documentation Co-authored-by: radical <1472+radical@users.noreply.github.com> * Complete CI improvements: Add OuterloopTestRunsheetBuilder, matrix-based workflows, and shared reusable workflow Co-authored-by: radical <1472+radical@users.noreply.github.com> * Fix outerloop workflow: Add test validation and handle empty runsheets - Add OuterloopTest attributes to resource-intensive tests for validation: - BuildAndRunAspireTemplate and BuildAndRunAspireTemplateWithCentralPackageManagement in Templates.Tests - VerifyComponentWorks integration tests in EndToEnd.Tests - ResourcesShowUpOnDashboard dashboard tests in StarterTemplateRunTestsBase - Fix Aspire.EndToEnd.Tests project to include TestUtilities reference - Update specialized-test-runner.yml to handle empty runsheets gracefully: - Skip run_tests job when runsheet is empty (no tests found) - Make results job conditional on whether tests were actually run - Prevent workflow failure when no outerloop tests exist The OuterloopTestRunsheetBuilder now successfully detects 9 outerloop tests across 4 configurations and generates proper runsheets for matrix-based execution. Co-authored-by: radical <1472+radical@users.noreply.github.com> * Fix OuterloopTest validation: Remove from EndToEnd/Templates, add to Dashboard/Hosting tests - Remove OuterloopTest attributes from EndToEnd and Templates tests as requested - Remove TestUtilities reference from EndToEnd.Tests project - Add OuterloopTest attributes to appropriate resource-intensive tests: - Dashboard Playwright browser tests (AppBarTests) - Dashboard integration startup tests (StartupTests) - Hosting Docker container tests (DistributedApplicationTests) - Add TestUtilities reference to Dashboard.Tests for OuterloopTest attribute support - Add required using statement for Aspire.TestUtilities in StartupTests.cs The OuterloopTestRunsheetBuilder now targets tests in more appropriate projects for validation rather than the core template and end-to-end tests. Co-authored-by: radical <1472+radical@users.noreply.github.com> * fix build * Update specialized workflows to use run-tests.yml and add requiresNugets field to runsheets Co-authored-by: radical <1472+radical@users.noreply.github.com> * fix use of run-tests.yml * Add OuterloopTest attributes to template tests for validation Co-authored-by: radical <1472+radical@users.noreply.github.com> * Use build-packages.yml workflow in specialized-test-runner.yml Co-authored-by: radical <1472+radical@users.noreply.github.com> * Add requiresTestSdk field to runsheet generation - set to true for template tests Co-authored-by: radical <1472+radical@users.noreply.github.com> * pass on requiresTestSdk * Optimize CI: Run generate_tests_matrix on ubuntu-latest and share code between test runsheet builders Co-authored-by: radical <1472+radical@users.noreply.github.com> * fix cmdline * Remove existing OuterloopTest attributes and add to RequiresPlaywright tests; make arch-specific nuget uploads conditional Co-authored-by: radical <1472+radical@users.noreply.github.com> * fix build * re-enable arch-specific nugets * build packages in parallel * Add workflow detection properties, fix Playwright installation control, and improve template test filtering Co-authored-by: radical <1472+radical@users.noreply.github.com> * fix up * correctly disable playwright testson the main run * fix condition * fix playwright supported * mark all playwright tests to run only on outerloop * cleanup * cleanup * address review feedback from @ danmoseley --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: radical <1472+radical@users.noreply.github.com> Co-authored-by: Ankit Jain <radical@gmail.com>
1 parent 2365bd5 commit 353aab4

24 files changed

+629
-359
lines changed

.github/copilot-instructions.md

Lines changed: 21 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -84,30 +84,30 @@
8484

8585
(1) Build from the root with `./build.sh` (~3-5 minutes).
8686
(2) If that produces errors, fix those errors and build again. Repeat until the build is successful.
87-
(3) To run tests for a specific project: `dotnet test tests/ProjectName.Tests/ProjectName.Tests.csproj --no-build -- --filter-not-trait "quarantined=true"`
87+
(3) To run tests for a specific project: `dotnet test tests/ProjectName.Tests/ProjectName.Tests.csproj --no-build -- --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"`
8888

8989
Note that tests for a project can be executed without first building from the root.
9090

9191
(4) To run specific tests, include the filter after `--`:
9292
```bash
93-
dotnet test tests/Aspire.Hosting.Testing.Tests/Aspire.Hosting.Testing.Tests.csproj -- --filter-method "TestingBuilderHasAllPropertiesFromRealBuilder" --filter-not-trait "quarantined=true"
93+
dotnet test tests/Aspire.Hosting.Testing.Tests/Aspire.Hosting.Testing.Tests.csproj -- --filter-method "TestingBuilderHasAllPropertiesFromRealBuilder" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
9494
```
9595

9696
**Important**: Avoid passing `--no-build` unless you have just built in the same session and there have been no code changes since. In automation or while iterating on code, omit `--no-build` so changes are compiled and picked up by the test run.
9797

98-
### CRITICAL: Excluding Quarantined Tests
98+
### CRITICAL: Excluding Quarantined and Outerloop Tests
9999

100-
When running tests in automated environments (including Copilot agent), **always exclude quarantined tests** to avoid false negatives:
100+
When running tests in automated environments (including Copilot agent), **always exclude quarantined and outerloop tests** to avoid false negatives and long-running tests:
101101

102102
```bash
103-
# Correct - excludes quarantined tests (use this in automation)
104-
dotnet test tests/Project.Tests/Project.Tests.csproj -- --filter-not-trait "quarantined=true"
103+
# Correct - excludes quarantined and outerloop tests (use this in automation)
104+
dotnet test tests/Project.Tests/Project.Tests.csproj -- --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
105105

106-
# For specific test filters, combine with quarantine exclusion
107-
dotnet test tests/Project.Tests/Project.Tests.csproj -- --filter-method "TestName" --filter-not-trait "quarantined=true"
106+
# For specific test filters, combine with quarantine and outerloop exclusion
107+
dotnet test tests/Project.Tests/Project.Tests.csproj -- --filter-method "TestName" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
108108
```
109109

110-
Never run all tests without the quarantine filter in automated environments, as this will include flaky tests that are known to fail intermittently.
110+
Never run all tests without the quarantine and outerloop filters in automated environments, as this will include flaky tests that are known to fail intermittently and long-running tests that slow down CI.
111111

112112
Valid test filter switches include: --filter-class, --filter-not-class, --filter-method, --filter-not-method, --filter-namespace, --filter-not-namespace, --filter-not-trait, --filter-trait
113113

@@ -140,7 +140,8 @@ Valid test filter switches include: --filter-class, --filter-not-class, --filter
140140

141141
### Continuous Integration
142142
- **`tests.yml`**: Main test workflow running across Windows/Linux/macOS
143-
- **`tests-outerloop.yml`**: Runs quarantined tests separately every 2 hours
143+
- **`tests-quarantine.yml`**: Runs quarantined tests separately every 6 hours
144+
- **`tests-outerloop.yml`**: Runs outerloop tests separately every 6 hours
144145
- **`ci.yml`**: Main CI workflow triggered on PRs and pushes to main/release branches
145146
- **Build validation**: Includes package generation, API compatibility checks, template validation
146147

@@ -158,13 +159,22 @@ Valid test filter switches include: --filter-class, --filter-not-class, --filter
158159

159160
- Tests that are flaky and don't fail deterministically are marked with the `QuarantinedTest` attribute.
160161
- Such tests are not run as part of the regular tests workflow (`tests.yml`).
161-
- Instead they are run in the `Outerloop` workflow (`tests-outerloop.yml`).
162+
- Instead they are run in the `Quarantine` workflow (`tests-quarantine.yml`).
162163
- A github issue url is used with the attribute
163164

164165
Example: `[QuarantinedTest("..issue url..")]`
165166

166167
- To quarantine or unquarantine tests, use the tool in `tools/QuarantineTools/QuarantineTools.csproj`.
167168

169+
## Outerloop tests
170+
171+
- Tests that are long-running, resource-intensive, or require special infrastructure are marked with the `OuterloopTest` attribute.
172+
- Such tests are not run as part of the regular tests workflow (`tests.yml`).
173+
- Instead they are run in the `Outerloop` workflow (`tests-outerloop.yml`).
174+
- An optional reason can be provided with the attribute
175+
176+
Example: `[OuterloopTest("Long running integration test")]`
177+
168178
## Snapshot Testing with Verify
169179

170180
* We use the Verify library (Verify.XunitV3) for snapshot testing in several test projects.

.github/workflows/run-tests.yml

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,11 @@ on:
3939
required: false
4040
type: boolean
4141
default: false
42+
# Controls whether to install Playwright browsers during project build
43+
enablePlaywrightInstall:
44+
required: false
45+
type: boolean
46+
default: false
4247
os:
4348
required: false
4449
type: string
@@ -205,8 +210,14 @@ jobs:
205210
if: ${{ ! inputs.requiresNugets }}
206211
env:
207212
CI: false
208-
run: |
209-
${{ env.BUILD_SCRIPT }} -restore -ci -build -projects ${{ env.TEST_PROJECT_PATH }} ${{ inputs.versionOverrideArg }}
213+
run: >
214+
${{ env.BUILD_SCRIPT }}
215+
-restore
216+
-ci
217+
-build
218+
-projects ${{ env.TEST_PROJECT_PATH }}
219+
${{ inputs.enablePlaywrightInstall == true && '' || '/p:InstallBrowsersForPlaywright=false' }}
220+
${{ inputs.versionOverrideArg }}
210221
211222
- name: Build and archive test project
212223
if: ${{ inputs.requiresNugets }}
@@ -216,6 +227,7 @@ jobs:
216227
${{ env.BUILD_SCRIPT }} -restore -ci -build -projects ${{ env.TEST_PROJECT_PATH }}
217228
/p:PrepareForHelix=true
218229
/bl:${{ github.workspace }}/artifacts/log/Debug/PrepareForHelix.binlog
230+
${{ inputs.enablePlaywrightInstall == true && '' || '/p:InstallBrowsersForPlaywright=false' }}
219231
${{ inputs.versionOverrideArg }}
220232
221233
# Workaround for bug in Azure Functions Worker SDK. See https://github.com/Azure/azure-functions-dotnet-worker/issues/2969.
@@ -272,6 +284,7 @@ jobs:
272284
DCP_DIAGNOSTICS_LOG_FOLDER: ${{ github.workspace }}/testresults/dcp
273285
BUILT_NUGETS_PATH: ${{ github.workspace }}/artifacts/packages/Debug/Shipping
274286
NUGET_PACKAGES: ${{ github.workspace }}/nuget-cache
287+
PLAYWRIGHT_INSTALLED: ${{ inputs.enablePlaywrightInstall == true && 'true' || 'false' }}
275288
TEST_LOG_PATH: ${{ github.workspace }}/artifacts/log/test-logs
276289
TestsRunningOutsideOfRepo: true
277290
run: >
@@ -296,6 +309,7 @@ jobs:
296309
# In this step, we are not using Arcade, but want to make sure that MSBuild is able to evaluate correctly.
297310
# So, we manually set NUGET_PACKAGES
298311
NUGET_PACKAGES: ${{ github.workspace }}/.packages
312+
PLAYWRIGHT_INSTALLED: ${{ inputs.enablePlaywrightInstall == true && 'true' || 'false' }}
299313
run: >
300314
${{ env.DOTNET_SCRIPT }} test ${{ env.TEST_PROJECT_PATH }}
301315
/p:ContinuousIntegrationBuild=true
Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# Reusable workflow for running specialized test suites (quarantined and outerloop tests)
2+
name: Specialized Test Runner
3+
4+
on:
5+
workflow_call:
6+
inputs:
7+
testRunnerName:
8+
required: true
9+
type: string
10+
description: 'Arcade test runner name'
11+
extraRunSheetBuilderArgs:
12+
required: false
13+
type: string
14+
extraTestArgs:
15+
required: false
16+
type: string
17+
# Controls whether to install Playwright browsers during project build
18+
enablePlaywrightInstall:
19+
required: true
20+
type: boolean
21+
default: false
22+
23+
jobs:
24+
25+
generate_tests_matrix:
26+
name: Generate test runsheet
27+
runs-on: ubuntu-latest
28+
if: ${{ github.repository_owner == 'dotnet' }}
29+
outputs:
30+
runsheet: ${{ steps.generate_tests_matrix.outputs.runsheet }}
31+
steps:
32+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
33+
34+
# We need to build the whole solution, so that we can interrogate each test project
35+
# and find out whether it contains any tests of the specified type.
36+
- name: Build the solution
37+
run: >
38+
./build.sh
39+
--restore
40+
--build
41+
-c Release
42+
--ci
43+
/p:CI=false
44+
/p:GeneratePackageOnBuild=false
45+
/p:InstallBrowsersForPlaywright=false
46+
47+
- name: Generate test runsheet
48+
id: generate_tests_matrix
49+
run: |
50+
./build.sh
51+
--test
52+
/p:TestRunnerName=${{ inputs.testRunnerName }}
53+
${{ inputs.extraRunSheetBuilderArgs }}
54+
-c Release
55+
--ci
56+
/p:CI=false
57+
/p:Restore=false
58+
/p:Build=false
59+
/bl:${{ github.workspace }}/artifacts/log/Release/runsheet.binlog
60+
61+
- name: Upload logs, and test results
62+
if: ${{ always() }}
63+
uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
64+
with:
65+
name: logs-runsheet
66+
path: |
67+
${{ github.workspace }}/artifacts/log/*/runsheet.binlog
68+
${{ github.workspace }}/artifacts/log/*/TestLogs/**
69+
${{ github.workspace }}/artifacts/tmp/*/combined_runsheet.json
70+
retention-days: 5
71+
72+
build_packages:
73+
name: Build packages
74+
if: ${{ github.repository_owner == 'dotnet' }}
75+
uses: ./.github/workflows/build-packages.yml
76+
77+
run_tests:
78+
name: Test
79+
needs: [generate_tests_matrix, build_packages]
80+
strategy:
81+
fail-fast: false
82+
matrix:
83+
tests: ${{ fromJson(needs.generate_tests_matrix.outputs.runsheet) }}
84+
if: ${{ always() && !cancelled() && github.repository_owner == 'dotnet' && needs.generate_tests_matrix.outputs.runsheet != '[]' }}
85+
uses: ./.github/workflows/run-tests.yml
86+
with:
87+
testShortName: ${{ matrix.tests.project }}
88+
requiresNugets: ${{ matrix.tests.requiresNugets == true }}
89+
requiresTestSdk: ${{ matrix.tests.requiresTestSdk == true }}
90+
os: ${{ matrix.tests.os }}
91+
enablePlaywrightInstall: ${{ inputs.enablePlaywrightInstall }}
92+
extraTestArgs: ${{ inputs.extraTestArgs }}
93+
94+
results:
95+
if: ${{ always() && github.repository_owner == 'dotnet' }}
96+
runs-on: ubuntu-latest
97+
name: Final Results
98+
needs: [generate_tests_matrix, build_packages, run_tests]
99+
steps:
100+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
101+
102+
# Check if there were no tests to run
103+
- name: Check if tests were run
104+
id: check_tests
105+
run: |
106+
if [ "${{ needs.generate_tests_matrix.outputs.runsheet }}" = "[]" ]; then
107+
echo "No tests found for this test type"
108+
echo "tests_run=false" >> $GITHUB_OUTPUT
109+
else
110+
echo "tests_run=true" >> $GITHUB_OUTPUT
111+
fi
112+
113+
# get all the test logs artifacts into a single directory
114+
- uses: actions/download-artifact@cc203385981b70ca67e1cc392babf9cc229d5806 # v4.1.9
115+
if: steps.check_tests.outputs.tests_run == 'true'
116+
with:
117+
pattern: 'logs-*'
118+
path: ${{ github.workspace }}/artifacts/all-logs
119+
120+
# Organize the .trx files by OS
121+
- name: Organize test results by OS
122+
if: steps.check_tests.outputs.tests_run == 'true'
123+
shell: pwsh
124+
run: |
125+
$logDirectory = "${{ github.workspace }}/artifacts/all-logs"
126+
127+
# Create OS-specific directories
128+
New-Item -ItemType Directory -Path "${{ github.workspace }}/testresults/ubuntu-latest" -Force
129+
New-Item -ItemType Directory -Path "${{ github.workspace }}/testresults/windows-latest" -Force
130+
New-Item -ItemType Directory -Path "${{ github.workspace }}/testresults/macos-latest" -Force
131+
132+
# Find all .trx files
133+
$trxFiles = Get-ChildItem -Path $logDirectory -Filter *.trx -Recurse
134+
135+
# Copy each .trx file to the appropriate OS folder
136+
foreach ($trxFile in $trxFiles) {
137+
if ($trxFile.FullName -match "ubuntu") {
138+
Copy-Item -Path $trxFile.FullName -Destination "${{ github.workspace }}/testresults/ubuntu-latest/" -Force
139+
} elseif ($trxFile.FullName -match "windows") {
140+
Copy-Item -Path $trxFile.FullName -Destination "${{ github.workspace }}/testresults/windows-latest/" -Force
141+
} elseif ($trxFile.FullName -match "macos") {
142+
Copy-Item -Path $trxFile.FullName -Destination "${{ github.workspace }}/testresults/macos-latest/" -Force
143+
}
144+
}
145+
146+
- name: Generate test results summary
147+
if: steps.check_tests.outputs.tests_run == 'true'
148+
env:
149+
CI: false
150+
run: >
151+
${{ github.workspace }}/dotnet.sh
152+
run
153+
--project ${{ github.workspace }}/tools/GenerateTestSummary/GenerateTestSummary.csproj
154+
--
155+
${{ github.workspace }}/testresults
156+
--combined

0 commit comments

Comments
 (0)