Skip to content

Improve sanity tests stability#298253

Merged
dmitrivMS merged 8 commits intomainfrom
dev/dmitriv/sanity-tests
Feb 27, 2026
Merged

Improve sanity tests stability#298253
dmitrivMS merged 8 commits intomainfrom
dev/dmitriv/sanity-tests

Conversation

@dmitrivMS
Copy link
Copy Markdown
Contributor

@dmitrivMS dmitrivMS commented Feb 27, 2026

Capture and upload screenshots when errors occur.
Close app and browser even if test fails.
Improve waiting logic and add retries in flaky places (such as Extensions view).

@dmitrivMS dmitrivMS added this to the March 2026 milestone Feb 27, 2026
@dmitrivMS dmitrivMS self-assigned this Feb 27, 2026
Copilot AI review requested due to automatic review settings February 27, 2026 14:13
@dmitrivMS dmitrivMS added the engineering VS Code - Build / issue tracking / etc. label Feb 27, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to make the sanity (Playwright-based) UI tests more reliable and debuggable by replacing fixed delays with state-based waits and by capturing/publishing screenshots on failures in CI.

Changes:

  • Replace brittle waitForTimeout usage in UI flows with more deterministic waits (e.g., waiting for the command palette to close, waiting for search results/buttons).
  • Add failure screenshot capture support in TestContext, plus a CLI option (--screenshots-dir) to control where screenshots are written.
  • Update the Azure Pipelines sanity job to create a screenshots directory, pass it into the test run, and publish it as a pipeline artifact.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
test/sanity/src/uiTest.ts Improves UI interaction stability by using typed input and state-based waits instead of fixed sleeps.
test/sanity/src/main.ts Wires screenshotsDir into TestContext from CLI options (but currently misses -s alias support).
test/sanity/src/index.ts Adds --screenshots-dir, -s to the CLI help/options for the sanity test runner.
test/sanity/src/context.ts Captures a screenshot of the active Playwright page when a test fails.
build/azure-pipelines/common/sanity-tests.yml Creates/publishes a screenshots artifact and passes a screenshots directory to sanity test runs.
Comments suppressed due to low confidence (1)

test/sanity/src/main.ts:20

  • -s/--screenshots-dir is documented and used by the pipeline, but main.ts doesn't define an alias for -s. With the current minimist config, -s <path> will populate options.s instead of options['screenshots-dir'], so screenshotsDir will always be undefined when the short flag is used. Add {'screenshots-dir': 's'} to the alias map here (or switch callers to use --screenshots-dir).
const options = minimist(process.argv.slice(2), {
	string: ['commit', 'quality', 'screenshots-dir'],
	boolean: ['cleanup', 'verbose', 'signing-check', 'headless', 'detection'],
	alias: { commit: 'c', quality: 'q', verbose: 'v' },
	default: { cleanup: true, verbose: false, 'signing-check': true, headless: true, 'detection': true },
});

@dmitrivMS dmitrivMS marked this pull request as ready for review February 27, 2026 15:42
@vs-code-engineering
Copy link
Copy Markdown
Contributor

vs-code-engineering bot commented Feb 27, 2026

📬 CODENOTIFY

The following users are being notified based on files changed in this PR:

@lszomoru

Matched files:

  • build/azure-pipelines/common/sanity-tests.yml

@dmitrivMS dmitrivMS enabled auto-merge (squash) February 27, 2026 15:58
@dmitrivMS dmitrivMS merged commit a0a6cd8 into main Feb 27, 2026
30 of 31 checks passed
@dmitrivMS dmitrivMS deleted the dev/dmitriv/sanity-tests branch February 27, 2026 16:06
DonJayamanne pushed a commit that referenced this pull request Mar 2, 2026
@vs-code-engineering vs-code-engineering bot locked and limited conversation to collaborators Apr 13, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

engineering VS Code - Build / issue tracking / etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants