[Feature] Deterministic playwright tests across all machines and environments

[younesjallouli]

Hello team,

I want first to thank you for the amazing work on playwright. It's really having a spring phase and becoming the leading tool in e2e automation.
However, there is still a big issue in the test development phase. I call this issue the spray and pray situation. it touches all kinds of teams, from startups to enterprises in my opinion:

A lot of times tests are not deterministic. Even if we try as teams to make them as deterministic as possible, by controlling the data in the app under test for example, or by setting the browser resolution ... There are still situations where tests will succeed on a tester's machine, but fail on the colleague's machine.
Oftentimes they fail because playwright could not find some locators (even if they are there), or screenshots that could differ between machines ... Chrome on macOS may behave differently from Chrome on Linux, etc.
This simply breaks the confidence in the tests: If I develop a feature, write a test and make it pass locally, there is always a probability that I will break another developer's workflow, because that test will fail for them.

(PS: Btw this spray and pray issue is exactly what that tools like dagger.io is trying to solve for the CI/CD world)

For local playwright tests, the solution to this kind of problem is to use containers to avoid these differences between environments, as many have tried in the past:

• How to use codegen inside docker container? playwright-python#274
• [Question] how to run tests inside docker with headless:true, want to see the browser. #18224
• https://www.oddbird.net/2022/11/30/headed-playwright-in-docker/
• https://blog.logrocket.com/setting-headless-chrome-node-js-server-docker/

So far, even if testers have experimented this idea, NO one has really solved this problem, by running tests locally in docker across all team members, mainly because:
Major reason: Running tests in docker implies that playwright tooling won't work anymore. However, everybody wants to keep using playwright codegen, playwright inspector and the vscode extension during test development phase, these tools are too valuable.
Minor reason: Nobody wants to write install scripts that install XServer, Xquartz, VNC, or whatever tool depending on the user's machine (linux, macOS...) and handle this complexity in a dockerfile.

I think solving this problem will unlock a new level of testing and confidence for the industry.

[pavelfeldman]

Different parts of your message imply different mitigations, so I'll break it down.

Oftentimes they fail because playwright could not find some locators (even if they are there)

If this happens, it means that the test is not following the best practices, namely does not use web assertions. If the test is using them, not finding the locator is not possible, as long as the page gets the locator in question eventually. The point here is that if this happens, Docker won't help. It might reduce the flakiness of the test, but won't address it.

Or screenshots that could differ between machines

Visual regression testing (VRT) is its own world, and in that world, environment is everything. Producing consistent pixels requires consistent environment and that's where Docker is helpful.

This simply breaks the confidence in the tests: If I develop a feature, write a test and make it pass locally, there is always a probability that I will break another developer's workflow, because that test will fail for them.

If the test exercises racy behavior, it can even fail on the same machine. It is important to follow the best practices (use awaiting assertions instead of sampling the state of the page in a non-deterministic manner).

What I am trying to say is that Docker is not a solution for determinism, but could be a solution for the VRT.
If you are interested in keeping your inner loop in the host system (vscode extension), but running tests in docker, you could refer to #26482.

[dgozman]

The discussion seems stale, closing. If you still encounter problems, please file a new issue with a repro and link to this one.