[Bug]: Different projects with different dependencies do NOT run at the same time increasing time & underutilising resources (workers)

[garciallorente-fer]

Version

1.43.0

Steps to reproduce

I have a big time increase in my real-life project because of the way Playwright runs projects & projects dependencies.
It currently runs projects in phases, each phase contains projects that are free to go and waits until all of them are finished, but that waiting causes big total time increases.

This is a close example to my real-case scenario:

projects: [ { name: 'End2End-NonAuth' , testMatch: '**.nonAuth.**' }, { name: 'LOGIN-Test-setup', testMatch: '**login.setup.ts**' }, { name: 'End2End-Auth', testMatch: '**.e2e.**', use: { storageState: loggedInStatePath }, dependencies: ['LOGIN-TEST-SETUP'] } ]
As you can see here, Project 'End2End-NonAuth' can run all the tests at the same time as 'LOGIN-Test-setup'.

Project "End2End-NonAuth" has like 10 tests that take up to 60 seconds total time (because they run in parallel that is the max time one of them test takes) they do take their time because need to be end-to-end tests, like most people with have in their projects.

Project "LOGIN-Test-Setup" has JUST ONE test file, because this is test is the dependency a lot of tests have, to login first before running some end-to-end tests, as most of people would have in their project configurations. (it just takes like 20 seconds)

If I am using 50 workers (Playwright Testing) , when you try to run all these tests, this is the sequence:

1. Run 'End2End-NonAuth' & 'LOGIN-Test-setup' projects, ( but not 'End2End-Auth' yet, because it needs "login-test-setup")
2. 'LOGIN-Test-setup' project finishes in less than 20 seconds
3. But 'End2End-NonAuth' keeps running for 40 seconds more and 'End2End-Auth' Cannot start. Loosing 40 seconds of execution time for those tests, and having many workers available to use

This time just increases x3 times more with retries=2 , if one of the test fails, delaying massively the detection of a test error, which is what we want when something is breaking and the most important time efficiency we want to see.

Expected behavior

SOLUTION: Every phase should not wait for all the phase projects to finish, it should only wait until the dependency project is finished

All the workers are always working 100% of the time, with no time loss, as it only waits for the dependency project to finish,, instead of just waiting for the project that has the dependency to finish.

Actual behavior

Playwright currently runs projects in phases, each phase contains projects that are free to go and waits until all of them are finished.

There are workers free to use for other phases that are not being used, while waiting for the whole phase to finish, instead of just waiting for the project that has the dependency to finish. That can create a lot of total time increase.

Additional context

For more context please look at the previous bug that was not solved but still closed: #29714

Environment

System: Microsoft Playwright Testing (50 workers) with 2 retries & fully Parallel.

[mxschmitt]

Hey! Could you help bringing this issue down to a more real-world scenario / code which we can run locally? I tried to reproduce the behaviour which you are mentioning but was not able to.

This is the repository I've created: https://github.com/mxschmitt/playwight-issue-30296

Thanks!

[garciallorente-fer]

Thanks for the reply @mxschmitt , I have created a PR for you with a more real-world scenario / code which we you run locally npm test

mxschmitt/playwight-issue-30296#1

In this example we can see a 50% increase in time inefficiency

Project1: login test project (2s)
Project2: e2e-auth project (5s) > requires project1
Project3: e2e-nonAuth project (10s) > no dependencies

Real world scenario , it should take just 10s to run in total (project3 time)
But it takes 15s (project2+project3 times)