Skip to content

[Question] Missing snapshots not generated for all projects #19836

@Danphillipz

Description

@Danphillipz

When running visual regression with Playwright, it is failing to update all missing snapshots for each project defined in the configuration file (See configuration below).

import { type PlaywrightTestConfig, devices } from '@playwright/test';

import { baseConfig } from '../../playwright.config.base';

const config: PlaywrightTestConfig = {
  ...baseConfig,
  /* Run tests in files in parallel */
  fullyParallel: true,
  /* Fail the build on CI if you accidentally left test.only in the source code. */
  forbidOnly: !!process.env.CI,
  /* Retry on CI only */
  retries: process.env.CI ? 2 : 0,
  /* Reporter to use. */
  reporter: process.env.CI ? 'dot' : 'html',
  use: {
    ...baseConfig.use,
    screenshot: 'only-on-failure',
  }, 
  /* Configuration for visual regression checks */
  updateSnapshots: "missing",
  expect: {
    toHaveScreenshot: {
      threshold: 0.2,
      animations: "disabled"
    }
  },
  projects: [
    {
      name: 'chromium',
      use: {
        ...devices['Desktop Chrome'],
      },
    },

    {
      name: 'firefox',
      use: {
        ...devices['Desktop Firefox'],
      },
    },

    {
      name: 'webkit',
      use: {
        ...devices['Desktop Safari'],
      },
    },

    /* Test against mobile viewports. */
    {
      name: 'Mobile Chrome',
      use: {
        ...devices['Pixel 5'],
      },
    },
    {
      name: 'Mobile Safari',
      use: {
        ...devices['iPhone 12'],
      },
    },
  ],
};

export default config;

When running my tests (where no screenshots currently exist) only a small subset of the missing screenshots are actually updated. The other projects fail to have their missing screenshots captured as each tests fails with the message "Test was interrupted."
image
You can see that many tests ran for a few ms before being interrupted, or did not start at all
image

However if I set the updateSnapshots value to "all" and again, delete all of the screenshots from the previous run so that we are in the same starting state, then all 10 screenshots get written as expected
image

I would expect that updateSnapshots value being set to "missing" would still run each project I have defined and capture all 10 screenshots, however this does not seem to be the case.

Is this possibly a bug, or have I missed something obvious?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions