Skip to content

fix(vscode): Reduce E2E Test Flakiness#9161

Merged
lambrianmsft merged 3 commits into
Azure:mainfrom
lambrianmsft:lambrian/vscode-e2e-resiliency
May 12, 2026
Merged

fix(vscode): Reduce E2E Test Flakiness#9161
lambrianmsft merged 3 commits into
Azure:mainfrom
lambrianmsft:lambrian/vscode-e2e-resiliency

Conversation

@lambrianmsft
Copy link
Copy Markdown
Contributor

@lambrianmsft lambrianmsft commented May 12, 2026 •
edited
Loading

Commit Type

  • feature - New functionality
  • fix - Bug fix
  • refactor - Code restructuring without behavior change
  • perf - Performance improvement
  • docs - Documentation update
  • test - Test-related changes
  • chore - Maintenance/tooling

Risk Level

  • Low - Minor changes, limited scope
  • Medium - Moderate changes, some user impact
  • High - Major changes, significant user/system impact

What & Why

Stabilizes the VS Code workspace conversion-create E2E flow that was failing in CI. The test now fills wizard inputs by label instead of index, waits for path/name validation before advancing, targets the exact Create workspace submit action, and hardens label/input lookup against transient webview re-renders with better diagnostics.

Impact of Change

  • Users:
    None
  • Developers:
    VS Code E2E CI coverage is more stable and easier to diagnose when conversion-create navigation fails.
  • System:
    Test-only change; no runtime, production, workflow designer, or customer-facing VS Code extension behavior changes.

Test Plan

  • Unit tests added/updated
  • E2E tests added/updated
  • Manual testing completed
  • Tested in: npx biome check --write apps\\vs-code-designer\\src\\test\\ui\\workspaceConversionCreate.test.ts; cd apps\\vs-code-designer; npx tsup --config tsup.e2e.test.config.ts; E2E_MODE=conversioncreateonly node src\\test\\ui\\run-e2e.js

Contributors

Screenshots/Videos

lambrianmsft and others added 2 commits May 11, 2026 20:29
@lambrianmsft
Harden conversion create E2E flow
91a4129 
Fill the convert-to-workspace wizard by label instead of positional inputs and wait for validation before advancing to review. Add diagnostics and safer Next-click fallbacks so the test fails with actionable state instead of silently mixing field values.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lambrianmsft
Target conversion create submit action
d52a517 
Use the exact Create workspace footer button and a single DOM click so the review-step submit action is not confused with the Review + create stepper button or blocked by VS Code notifications. Keep the post-click assertion focused on the actual submit button state and workspace file creation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 12, 2026 14:35
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 12, 2026 •
edited
Loading

🤖 AI PR Validation Report

PR Review Results

Thank you for your submission! Here's detailed feedback on your PR title and body compliance:

⚠️ PR Title

  • Current: fix(vscode): Reduce E2E Test Flakiness
  • Issue: The title uses a fix(...) prefix but the PR body selects test as the Commit Type. This is a minor consistency issue only — the change is test-only and low risk. The title is otherwise descriptive and clear about intent.
  • Recommendation: Prefer aligning the conventional prefix with the PR body. Example: test(vscode): Reduce E2E test flakiness (stabilize workspace conversion-create flow) or test(vscode): Stabilize workspace conversion-create E2E flow

✅ Commit Type

  • Properly selected (test).
  • Note: Only one option is selected which is correct and matches the nature of the changes (test-only changes to an existing E2E test).

✅ Risk Level

  • The PR body marks Low risk and the repo labels include risk:low.
  • Assessment: The code diff shows modifications only to a single E2E test file (apps/vs-code-designer/src/test/ui/workspaceConversionCreate.test.ts). The changes are test-only and add resilience/diagnostics; advised risk: low (matches label).

✅ What & Why

  • Current: "Stabilizes the VS Code workspace conversion-create E2E flow that was failing in CI..."
  • Issue: None — the description is brief, clear, and explains why the changes were made.
  • Recommendation: None required; it's a good, concise explanation.

✅ Impact of Change

  • The body clearly states there are no user-facing or production runtime changes and describes developer/test benefits.
  • Recommendation: Keep as-is.
    • Users: None
    • Developers: VS Code E2E CI coverage is more stable
    • System: Test-only changes

✅ Test Plan

  • The PR body indicates E2E tests were updated and shows the test path and commands used to run them.
  • Code diff confirms updates to the E2E test file. This meets the requirement to validate that claimed tests exist.
  • Recommendation: None required. If you have CI logs showing flakiness improvement, consider linking them in a follow-up comment for reviewers.

⚠️ Contributors

  • Assessment: Empty. It's fine to be blank, but it's often helpful to credit reviewers/PMs/designers who contributed.
  • Recommendation: Optional — add other contributors if applicable.

⚠️ Screenshots/Videos

  • Assessment: Not applicable for test-only changes.
  • Recommendation: None — no visual changes.

Summary Table

Section Status Recommendation
Title ⚠️ Align conventional prefix with PR body (test vs fix)
Commit Type ✅ No change needed
Risk Level ✅ No change needed (advised risk: low)
What & Why ✅ No change needed
Impact of Change ✅ No change needed
Test Plan ✅ No change needed
Contributors ⚠️ Optional: add contributors if any
Screenshots/Videos ⚠️ Not applicable for test-only change

Final message
The PR passes the PR-body/template checks. The change is test-only (E2E) and limited to one test file — advised risk is low, which matches the risk:low label on the PR. The only actionable suggestion is to make the PR title and the Commit Type consistent (either change the title prefix from fix(...) to test(...) or update the commit type in the PR body if you intentionally want fix). Optionally add contributors if relevant. Thank you — this looks ready to merge after you optionally adjust the title for consistency.

Last updated: Tue, 12 May 2026 18:13:18 GMT

Copilot AI reviewed May 12, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the VS Code workspace-conversion UI E2E test to reduce flakiness by making form-filling, readiness checks, and wizard navigation more deterministic and adding richer diagnostics when the wizard gets stuck.

Changes:

  • Adds robust element-finding and input-typing helpers (exact button text matching, labeled input discovery, input value settling).
  • Replaces index-based form filling with label-driven filling and adds an explicit “setup ready” wait before navigating to the review step.
  • Improves “Next” navigation reliability with multiple click/focus fallbacks and enhanced diagnostics/screenshot capture on failure.

Comment thread apps/vs-code-designer/src/test/ui/workspaceConversionCreate.test.ts Outdated
Comment thread apps/vs-code-designer/src/test/ui/workspaceConversionCreate.test.ts
@lambrianmsft
Handle stale labels in conversion create E2E
5283d8a 
Guard follow-up label reads and container traversal in findInputByLabel so transient webview re-renders do not fail the conversion-create test with stale element errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lambrianmsft lambrianmsft added the risk:low Low risk change with minimal impact label May 12, 2026
@lambrianmsft lambrianmsft merged commit 6451c49 into Azure:main May 12, 2026
21 checks passed
lambrianmsft added a commit to lambrianmsft/LogicAppsUX that referenced this pull request May 12, 2026
@lambrianmsft
docs(squad): add chronicle-driven improvement playbook and session le…
3665568 
…arnings

- Add .squad/playbooks/chronicle-driven-improvement.md and matching
  prompt template .squad/prompts/chronicle-improve.md so /chronicle
  reindex + /chronicle improve feed into session-knowledge-curator
  as advisory candidate input only.
- Add session-knowledge-curator source bullets for chronicle in the
  charter and session-knowledge-feed playbook.
- Add durable learnings to .squad/knowledge/vscode-e2e-testing.md
  (label-driven fields, exhaustive stale-element handling,
  validation-gated Next/Create, exact-text button targeting,
  shared DOM helpers) and .squad/knowledge/review-patterns.md
  (scope-deferred reviewer reply pattern) sourced from PR Azure#9161.
- Document the chief-engineer subagent-consultation rule and the
  chronicle workflow in .squad/knowledge/agent-improvements.md.
- Point .squad/knowledge/README.md at the new playbook/prompt.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lambrianmsft added a commit to lambrianmsft/LogicAppsUX that referenced this pull request May 12, 2026
@lambrianmsft
Merge agent-dev into lambrian/codeful_experience_privatepreview
075af68 
Brings the Squad chief-engineer agent system, VS Code E2E flakiness
fixes (PR Azure#9161 lineage), and the chronicle-driven improvement
playbook into the codeful private preview branch so we can run
using the new agents.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot mentioned this pull request May 14, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@takyyon takyyon takyyon approved these changes

Assignees

No one assigned

Labels

pr-validated risk:low Low risk change with minimal impact

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@lambrianmsft @takyyon