[Feature]: Add option to record traces/screenshots/videos when snapshot tests run with --update-snapshots

[lorenz-gruber-dt]

🚀 Feature Request

Scenario
I'm running visual regression tests in my pipeline with the --update-snapshots flag enabled, so screenshots are automatically updated and can be committed by the pipeline in case they change.

Problem
I would like to record a trace if a test case fails and updates its snapshot. However, since snapshot test cases don't technically fail with --update-snapshots enabled, Playwright doesn't record any traces when using the retain-on-failure mode.

As far as I'm aware my only options at the moment are to either:
A. Use the trace mode on, which isn't ideal since then every test case always records a trace.
B. Run my tests first without the --update-snapshots flag to get a trace for failing test cases, and then run them again with the flag enabled to get updated screenshots. This approach would be better than always recording traces, but it adds unnecessary complexity to the pipeline IMO.

Feature Proposal
My proposal to solve this would be to either:
A. Adapt the retain-on-failure trace mode to also count updating a snapshot as a failure.
B. Introduce a new trace mode, e.g. retain-on-snapshot-change for this specific use case.

Example

In playwright.config.ts:

use: {
  trace: {
    mode: 'retain-on-snapshot-change',
  }
}

Motivation

I think this would be a valid addition to Playwright, as it makes it simpler to set up snapshots tests without requiring complicated workarounds to also leverage tracing.

[dgozman]

@lorenz-gruber-dt In your scenario, Playwright does not know in advance whether a snapshot would be updated, so the trace will have to be recorded for every test anyway. The only difference between 'on' and 'retain-on-snapshot-change' is that some traces will be recorded and discarded when using the latter.

I would recommend a modified option B for you:

• run all tests;
• rerun failures with --update-snapshots and --trace=on.
  This way you will only record the traces you need.

If you are fine with recording traces for every test:

• run all tests with --update-snapshots and --trace=on;
• only upload/check the traces you are interested in.

Let me know whether this works for you.