Re-enable VS Code extension E2E tests in CI - #19142
Draft
Adam Ratzman (adamint) wants to merge 3 commits into
Draft
Re-enable VS Code extension E2E tests in CI#19142Adam Ratzman (adamint) wants to merge 3 commits into
Adam Ratzman (adamint) wants to merge 3 commits into
Conversation
github-actions
Bot
added
the
area-engineering-systems
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 19142Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 19142" |
Contributor
There was a problem hiding this comment.
Pull request overview
Re-enables VS Code extension E2E tests and restores aggregate-gate enforcement.
Changes:
- Removes the unconditional skip guard.
- Treats unexpectedly skipped E2E runs as failures.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/tests.yml |
Restores E2E execution and gate checks. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Balanced
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
There was a problem hiding this comment.
Review details
Suppressed comments (2)
.github/workflows/extension-e2e-tests.yml:258
- The temporary-disable behavior has no regression coverage even though
extension/src/test/e2eLaunchProfile.test.ts:216-236already source-checks this exact Azure Functions shard. Without assertingdisabledIssueand the prerequisite guard, a later edit can silently re-enable issue #19151 while those tests still pass. Please update that test to pin the disabled marker and guarded install behavior.
disabledIssue: 'https://github.com/microsoft/aspire/issues/19151'
.github/workflows/tests.yml:830
- This comment does not match the current selector configuration.
setup_for_testspassesenforce: 'false'at tests.yml:69, and.github/actions/select-tests/action.yml:19documents that audit mode sets everyrun_*output to true; non-PR events also force the full set. Therefore this job currently runs on unrelated changes too. Describe the condition in terms of the selector output so the aggregate-gate exception remains accurate in both audit and enforce modes.
# - extension_e2e_tests: this job only runs when the PR/push changes the VS Code extension,
# Aspire CLI, or the extension E2E workflow wiring.
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Balanced
8 tasks
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
There was a problem hiding this comment.
Review details
Suppressed comments (1)
extension/src/test/e2eLaunchProfile.test.ts:510
- This lookup resolves the earlier
waitForWorkspaceRediscoveryLoadingcall at line 92, not the call in the gated test. Removing or moving the gated test's loading assertion afterreleasePsSnapshot()would therefore still pass this regression test and allow the CI race to return. Match the unique call and verify every index exists before checking their order.
assert.ok(appHostTree.indexOf('await waitForWorkspaceRediscoveryLoading') < appHostTree.indexOf('discoveryGate.releasePsSnapshot();'));
- Files reviewed: 5/5 changed files
- Comments generated: 0 new
- Review effort level: Balanced
Reverts the `false &&` guard added in microsoft#18547 and restores the two run_extension_e2e skip-checks in the aggregate gate. The job has been dark since 2026-06-29. microsoft#18547 disabled it because the Windows debug-dashboard shard was consistently failing, but it linked microsoft#18412 as the tracker -- and that issue had already been closed as completed on 2026-06-26 by the *previous* re-enable (microsoft#18464). So there is currently no open issue tracking the disable, and nothing surfaces that the shards do not run. This change is a probe: CI on this PR exercises the shards against current main so we can see empirically whether debug-dashboard still fails, rather than inferring it from a six-week-old report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 487b899d-d79d-4349-a84b-a026861bde7f
The re-enable probe showed the non-Functions VS Code extension E2E shards passing, while the Linux Azure Functions shard timed out waiting for e2e-functions to reach Running. Keep the shard visible in the workflow, but leave the Functions fixture disabled with a tracked issue until the Linux startup timeout is root-caused. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Gate the simulated ps and streaming ls CLI calls so the test observes the loading welcome before allowing the running AppHost snapshot and workspace candidate to arrive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adam Ratzman (adamint)
force-pushed
the
reenable-extension-e2e
branch
from
August 7, 2026 20:10
aada9c2 to
7a481aa
Compare
Contributor
There was a problem hiding this comment.
Review details
Suppressed comments (1)
extension/src/test/e2eLaunchProfile.test.ts:510
- This lookup matches the earlier unrelated loading wait in
appHostTree.e2e.test.ts(currently line 92), not the wait in the gated test. As a result, the assertion still passes ifreleasePsSnapshot()is moved before the intended wait, so the regression guard does not enforce the ordering it claims to protect. Scope the indices to the gated test block.
assert.ok(appHostTree.indexOf('await waitForWorkspaceRediscoveryLoading') < appHostTree.indexOf('discoveryGate.releasePsSnapshot();'));
- Files reviewed: 5/5 changed files
- Comments generated: 0 new
- Review effort level: Balanced
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-enables the VS Code extension E2E workflow by removing the
false &&guard and restoring the aggregate-gate skip checks.Probe result
Run 31202231991 exercised the E2E matrix after the guard was removed.
Verify ExTester internal feed availabilitypassed.Passed on both Linux and Windows in that run:
apphost-tree,command-palette,debug-dashboard,debug-startup-timeout,discovery-configuration,edge-cases,package-surface,settings-files,tree-actions, andzero-to-running.debug-dashboardpassing on both platforms removes the main known reason the job had been disabled.A later run exposed a race in
apphost-tree: the test waited for the transientSearching for AppHosts...welcome after discovery could already finish. This PR now gates the simulatedpsand streaminglsCLI calls so the loading state is observed deterministically before releasing the running AppHost snapshot and then the workspace candidate.The only deterministic E2E shard failure from the original probe was
azure-functionson Linux: it timed out after 300000ms waiting fore2e-functionsto reachRunningwithLast error: <none>. The job log shows Azure Functions Core Tools 4.12.1 was installed andfunc --versionsucceeded, so this was not a missing Core Tools install. The workflow currently has no Windowsazure-functionsshard.Follow-up
The
azure-functionsshard remains visible in the reusable workflow, but the Functions fixture is disabled pending #19151. The other VS Code extension E2E shards are re-enabled.Validation after the AppHost race fix: run 31212569613 passed both
apphost-treejobs (LinuxandWindows) and reported no VS Code extension E2E failures.