E2Ev2

[RXminuS]

We've been plagued by slow and flaky e2e tests. And although Agent tests are effective in eliminating some of that flake they lack the ability to validate real extension behavior and UI states.

The new E2Ev2 framework brings together the best elements of the Agent Tests with a more Developer-Friendly E2E workflow. The initial goal of this framework is to provide the baseline tools to migrate the flaky E2Ev1 tests and re-enable them in CI.

But the ultimate goal is to make the DX fast enough to serve as the go-to way to replicate issues, interactively develop new UI features (watch mode), or fuzzy-test core features across a wide variety of contexts.

Highlights

• significantly faster
  • all tests can run in parallel
  • all tests can run headless and only require a lightweight VSCode-Server process
  • all http is pre-recorded and played back with 0 delays (but can be controlled per request if needed)
• more readable & maintainable
  • abstract common functionality in UIX helpers
  • allows the calling of VSCode commands
  • You can invoke custom macros that have full access to the vscode-api. This makes it trivial to set-up testing conditions without requiring tricky to time UI state
• Flake is fully eliminated
  • all tests are 100% isolated
  • all http is recorded and can be optionally mocked for custom responses
  • timing of http responses can be controlled. This is great for being able to try things like aborting requests even thought the http is recorded
• More dev friendly & #velocity-wins
  • you can configure every parameter of the frameworks including things like installed extensions, vscode version, user/workspace settings etc.
  • You can use playwright to auto-record your tests
  • You can leverage the excellent playwright --ui and --watch flags to create iterative dev workflows
  • Testing output in CI will contain full logs, network traces, interactive browser recording (much better than video as you can still use your chrome-devtools to inspect stuff)
  • Tests are structured with steps that make the event log more interpretable

Test Plan

This framework should not interfere or break any existing functionality as it's not currently enabled.

• Existing tests pass
• Manually verified that builds work across platforms

[RXminuS]

• E2Ev2 #5278 : 2 dependent PRs (#5026 , #5558 ) 👈
• Status Bar: Observable Migration #5757
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @RXminuS and the rest of your teammates on Graphite

[RXminuS]

@keegancsmith I've merged a whole stack of branches into one because most of the early PRs had the majority of their code tweaked or removed. This now feels like a good enough starting point to at least have been able to migrate 1 flaky test :-)

Would love to find some time to pair and migrate a few more together?

[dominiccooney]

Some incremental feedback inline.

[dominiccooney]

??? what does this comment relate to?

[keegancsmith]

Would love to find some time to pair and migrate a few more together?

Lets pair monday morning?