[Feature]: Dynamic `webServer` configuration based on worker count

[reteps]

🚀 Feature Request

Summary

Currently, the webServer configuration is a static object. However, tests have access to a TEST_WORKER_INDEX value. This means that while tests are aware of what shard they are in, there is no way to give each shard a separate web server. Doing so would make test isolation easier.

Ideally, we could spin up the correct number of web servers based on how many workers we need.

At boot time, our web server creates and seeds a test database, the name of which is configurable. This works great when we have a single worker. When we have multiple workers running on the same machine, the web server is not aware of the worker information on boot.

Thus, we share a single web server between all workers. However, certain tests we have set the database into a certain state before performing browser tests. This causes issues when multiple tests run concurrently because database state leaks between workers, causing flaky/failing tests. Consider the following scenario:

testA: worker 1
- creates a new user, and goes to the user listing page to assert the count of users is 1. Then, cleans up the database.
testB: worker 2
- creates a new user, and asserts that they can login. Then, cleans up the database.

If the tests run concurrently, there is a scenario in which on the user listing page could show the count of users is 2, failing the test.

Ideally, we can spin up a separate webserver for each worker. Playwright already allows creating multiple webservers, but this configuration is static and AFAIK no information about the worker setup is available when this is booted up.

I propose that:

1. the webServer property could be a function that returns a configuration object, instead of just the configuration object
2. This function accepts information about the test setup that can be used to modify the configuration

Example

Here is how we could use this in our project:

{
  webServer: ({ workerCount }) => {
    return Array.from(Array(workerCount).keys()).map(idx => ({
      command: `yarn run dev --worker=${idx}`,
      url: `http://localhost:${3000 + workerIndex}`,
      reuseExistingServer: !process.env.CI,
      stdout: 'pipe',
      stderr: 'pipe',
    })
},

• The web server can connect to a different test database based on the worker field.
• In tests, we can connect to http://localhost:${3000 + process.env.TEST_WORKER_INDEX}.

Motivation

We want to have multiple highly-parallel workers, but we still need the isolation between workers. This offers us an easy way to do that.

[pavelfeldman]

That's what worker fixtures are for. If you'd like to have a web server per worker (that's what we do for isolation), get yourself a worker fixture that starts and stops the server.

[reteps]

Thanks, I didn't think of that! That solves my issue.