[Bug]: Testing stopped early after ${x} maximum allowed failures do not throw error. This makes pipeline succesed when tests have been stoped early by playwright.

[scharfi49]

Version

1.42.1-jammy

Steps to reproduce

Unfortunately i cannot introduce 1:1 steps to generate this bug to happens. It feels its happens once per multiple pipelines randomly, im using Jenkins to create docker container where tests are executed.
Pipeline logs that can help and i can share with u:
log.txt

Update:
After scanning multiple pipelines it occurs to happen on "testing stopped early after ${x} maximum allowed failures" event -> this event do not fire error.

Expected behavior

On: testing stopped early after ${x} maximum allowed failures, i should get error. To update my pipeline: "some tests do not run at all because of maximum allowed failures has been reached."

Actual behavior

 Slow test file: [chromium] › e2e/02-authed/08-settings/05-dashboard/04-automations-widget/happy-path.spec.ts (1.5m)
  Slow test file: [chromium] › e2e/02-authed/04-automations/happy-path.spec.ts (1.3m)
  Slow test file: [chromium-setup] › e2e/01-preAuthed/01-sign-up/01-validation.spec.ts (45.4s)
  Slow test file: [chromium] › e2e/02-authed/02-home/01-happy-path.spec.ts (44.5s)
  Slow test file: [chromium] › e2e/02-authed/08-settings/03-profile/01-data/changeLanguage.spec.ts (44.0s)
  Consider splitting slow test files to speed up parallel execution
Testing stopped early after 6 maximum allowed failures.
  4 flaky
    [chromium] › e2e/02-authed/04-automations/happy-path.spec.ts:49:7 › Automations › Should show page correctly 
    [chromium] › e2e/02-authed/04-automations/happy-path.spec.ts:129:7 › Automations › Automation should not print data on console 
    [chromium] › e2e/02-authed/08-settings/05-dashboard/04-automations-widget/happy-path.spec.ts:35:11 › Automations Widget › Automations widget functionality › Should add widget correctly 
    [chromium] › e2e/02-authed/08-settings/05-dashboard/04-automations-widget/happy-path.spec.ts:80:11 › Automations Widget › Cleaning › Should show warning on widget after scenes deletion 
  5 skipped
  188 did not run
  136 passed (12.5m)

To open last HTML report run:

  npx playwright show-report playwright-report/chromium

There is no error after "Testing stopped early after 6 maximum allowed failures."
Pipeline is succeed when 188 tests did not run.

Additional context

Current config:

import { PlaywrightTestConfig, devices } from '@playwright/test'

const NUM_RETRIES = 2
const MAX_FAILURES = NUM_RETRIES * 3

export const makeConfig = (
  browser: 'chromium' | 'firefox' | 'webkit' | 'iPhone8' | 'pixel7',
): PlaywrightTestConfig => {
  const d = {
    chromium: 'Desktop Chrome',
    firefox: 'Desktop Firefox',
    webkit: 'Desktop Safari',
    iPhone8: 'iPhone 8',
    pixel7: 'Pixel 7',
  }

  const supportSnaps = ['chromium', 'pixel7']

  //@See why: https://github.com/microsoft/playwright/issues/11627
  const launchOptions = {
    args:
      browser !== 'webkit' && browser !== 'iPhone8'
        ? ['--ignore-gpu-blocklist', '--use-gl=angle', '--use-angle=gl-egl']
        : [],
  }

  return {
    testDir: `./playwright`,
    outputDir: `./test-results/${browser}`,
    ignoreSnapshots: !supportSnaps.includes(browser),
    fullyParallel: false,
    forbidOnly: !!process.env.JENKINS_CI,
    retries: process.env.JENKINS_CI ? NUM_RETRIES : 0,
    workers: 1,
    maxFailures: process.env.JENKINS_CI ? MAX_FAILURES : undefined,
    timeout: 60 * 1000 * 1,
    expect: {
      timeout: 30000,
      toHaveScreenshot: {
        maxDiffPixels: 500,
      },
    },
    use: {
      baseURL: process.env.CLIENT_URL,
      trace: 'retain-on-failure',
      video: 'on',
      ignoreHTTPSErrors: true,
      ...devices[d[browser]],

      /**
       * @todo This is a workaround for the issue with phone agents that doesn't being able to
       * fire touch event. In future we should investigate synthetic events in
       * our fork. It happen only test so that's why we didn't create a bug.
       * @see https://github.com/TechSterowniki/react-repeatable
       * @see https://dev.azure.com/TECHSterowniki/eHome/_workitems/edit/13837
       */
      ...(browser === 'iPhone8' || browser === 'pixel7'
        ? {
            userAgent: devices['Desktop Chrome'].userAgent,
          }
        : {}),
    },
    reporter: [
      ['html', { outputFolder: `playwright-report/${browser}`, open: 'never' }],
    ],
    projects: [
      ...(!process.env.SKIP_REGISTER
        ? [
            {
              name: `${browser}-setup`,
              testDir: './playwright/e2e/01-preAuthed',
              use: { launchOptions },
            },
          ]
        : []),
      {
        name: `${browser}-auth`,
        testMatch: /.*\.setup\.ts/,
        dependencies: !process.env.SKIP_REGISTER ? [`${browser}-setup`] : [],
      },
      {
        ...(process.env.TEST_MATCH && {
          testMatch: new RegExp(process.env.TEST_MATCH),
        }),
        name: browser,
        testDir: './playwright/e2e/02-authed',
        use: {
          storageState: `./playwright/.auth/user-${browser}-auth.json`,
          launchOptions,
        },
        dependencies: [`${browser}-auth`],
      },
    ],
  }
}

Environment

# npx envinfo --preset playwright

  System:
    OS: Linux 6.4 Ubuntu 22.04.3 LTS 22.04.3 LTS (Jammy Jellyfish)
    Memory: 9.86 GB / 11.69 GB
    Container: Yes
  Binaries:
    Node: 20.11.1 - /usr/bin/node
    Yarn: 3.6.0 - /usr/bin/yarn
    npm: 10.2.4 - /usr/bin/npm
  Languages:
    Bash: 5.1.16 - /usr/bin/bash