[Bug]: Project dependency and storageState not working as expected

[alectrocute]

Version

1.43.1

Steps to reproduce

1. Create a new project
2. Configure a setup project with a testMatch of /.*\.setup\.ts/
3. Configure a Desktop Chrome project with a defined use.storageState and a ["setup"] as dependencies
4. Run a single test using Desktop Chrome, observe the setup dependency run, the storage state file is generated and the single test is ran
5. Run a single test again, observe dependency running yet again, overwriting the storage state file

Expected behavior

I'd expect the dependency to run once and generate the storage state file. And then in subsequent runs, the dependency should not be run since the storage state file already exists. If I play all the tests, eg. using the button, then the expected behavior occurs, eg. the setup only runs once before all the tests.

Actual behavior

The dependency test runs before every single test when running a single test.

Additional context

No response

Environment

System:
    OS: macOS 14.4.1
    CPU: (10) arm64 Apple M1 Pro
    Memory: 5.66 GB / 32.00 GB
  Binaries:
    Node: 18.15.0 - ~/.nvm/versions/node/v18.15.0/bin/node
    npm: 9.5.0 - ~/.nvm/versions/node/v18.15.0/bin/npm
  IDEs:
    VSCode: 1.88.1 - /usr/local/bin/code
  Languages:
    Bash: 3.2.57 - /bin/bash
  npmPackages:
    @playwright/test: ^1.43.1 => 1.43.1

[yury-s]

I'd expect the dependency to run once and generate the storage state file. And then in subsequent runs, the dependency should not be run since the storage state file already exists.

You can easily achieve that by having you setup code check if the storage state file already exists and skip expensive setup steps in that case. Playwright does not do this automatically.

const authFile = 'playwright/.auth/user.json';

setup('authenticate', async ({ request }) => {
  if (fs.existsSync(authFile))
    return;
  // Send authentication request. Replace with your own.
  await request.post('https://github.com/login', { ... });
  await request.storageState({ path: authFile });
});

If I play all the tests, eg. using the Screenshot 2024-04-18 at 8 16 41 AM button, then the expected behavior occurs, eg. the setup only runs once before all the tests.

Playwright will run setup dependencies on every run. It will run each dependency exactly once, regardless of how many tests are in the dependent project.

[alectrocute]

You can easily achieve that by having you setup code check if the storage state file already exists and skip expensive setup steps in that case. Playwright does not do this automatically.

Yeah, that's what I'm doing currently.

setup("Authenticate into CMS", async ({ page }) => {
  if (fs.existsSync(storageFilePath) && !fileOlderThan(storageFilePath, "4h")) {
    return;
  }
  ...
});

It would be nice if it did this automatically somehow, but I get it. The biggest downside with this is when running single tests with 'Show browser' enabled, it'll flash a white page in a new window before opening the actual test.

[yury-s]

The biggest downside with this is when running single tests with 'Show browser' enabled, it'll flash a white page in a new window before opening the actual test.

You can call test.skip based on a condition outside the tests:

setup.skip(fs.existsSync(storageFilePath) && !fileOlderThan(storageFilePath, "4h"));
setup("Authenticate into CMS", async ({ page }) => {
  ...
});

Or wrap relevant tests into test.describe and skip only it if you still want to run the rest.

[alectrocute]

setup.skip(fs.existsSync(storageFilePath) && !fileOlderThan(storageFilePath, "4h"));

@yury-s, that's... brilliant. This needs to be in the docs on the Authentication page.