[Feature]: Async test detection

[Jas0nch]

🚀 Feature Request

Hi Playwright team, is it possible for Playwright to support async test detection? The playwright did the test detection during the initial of the test run, so it can only accept fixed parameters.

For now, it's impossible to have the tests detected during runtime. So I'm asking if it's possible to add support for dynamic test detection in runtime?

Example

For example, if we have a page contains a dropdown with dynamic options, and we want to have a parallel parameterized test set against every option of that dropdown, it's not possible to do so.

Also, If we have a list of test data provided by some api, we are not able to test each of those test data but it can only be a fixed number located locally.

Motivation

• Supporting dynamic test detection can bring flexility and more capability to Playwright

[mxschmitt]

Its very unlikely that this is going to happen because Playwright's test-runner is designed to load tests in two phases. Test discovery in the beginning and test execution later on. We'd have to know during test discovery about your unknown amount of other tests you'd like to define but during that phase we don't have any fixtures/pages etc.

We usually recommend either using test.step for things like this or pre-defining it hard-coded e.g. how many dropdown items you want to test. This makes your test also much more assertive compared to having it dynamic, so you actually find out when the website changes.

Have you considered using test.step?

[Jas0nch]

As far as I know, test.step is synchronized so it may add extra time for the tests. But I could be wrong and please let me know if there’s a way to make it run in parallel.

[mxschmitt]

Yes, this means that it won't run in parallel.

How many drop-down items are you trying to test? Why is hard-coding them not an option for you?

[Skn0tt]

You should be able to do this:

const myBigListOfScenarios = await getTheDropdownOptions()
for (const scenario of myBigListOfScenarios) {
  test(scenario.name, async ({ page }) => {
    // execute test based on `scenario`
  });
}

As long as your version of Node.js supports top-level await, that's fine. This has the downside that test scanning potentially takes very long though, which isn't a scenario that Playwright is designed towards.

Alternatively, you could generate test files, write them to disk, and then execute them as usual.