[Question] Should timeout trigger retry?

[sb-michal-murray]

I think it does not trigger ATM. I think it should be expected behavior that it'd.

[mxschmitt]

Please provide us a repro and more details which timeout you mean and what your use-case is. Thanks!

[sb-michal-murray]

test('Slow test', async ({ page }) => {
  test.setTimeout(180_000)
  await doSomeStuff(page)
  await Promise.all([
      this.submit.click(),
      this.page.waitForNavigation({waitUntil: 'domcontentloaded' }),
    ])
})

where doSomeStuff() basically fills some form.

It tests a flaky site: what happened lately is that test waited for navigation to occur after submit click, but then page was closed due to time out.
logs:

 Timeout of 180000ms exceeded.
[05:03:56]	[Step 2/2] 
[05:03:56]	[Step 2/2]     page.waitForNavigation: Navigation failed because page was closed!
[05:03:56]	[Step 2/2]     =========================== logs ===========================
[05:03:56]	[Step 2/2]     waiting for navigation until "domcontentloaded"
[05:03:56]	[Step 2/2]     ============================================================

What I'd expect to happen is: first, that waitForNavigation() would timed-out before 180 s (should be 30s by default iirc), second, retry will be run (retries were set to 2 in config), but none of that happened

[chenrui333]

Yeah, I have run into the same question. Looks like my test did not get retried either (I use default timeout setting though).

[yury-s]

What I'd expect to happen is: first, that waitForNavigation() would timed-out before 180 s (should be 30s by default iirc),

timeout is per test. You can set action and navigation timeouts to custom values per test:

// playwright.config.js
const config = {
  ...
  timeout: 30_000,
  use: {
    navigationTimeout: 5_000
  }
}

second, retry will be run (retries were set to 2 in config), but none of that happened

This should work as long as global timeout is not reached:

// playwright.config.js
const config = {
  // Timeout per test
  timeout: 100,
  // If a test fails, retry it additional 2 times
  retries: 2,
}

With this config following test will be run 3 times.

const test = require('@playwright/test');
test('my test', async ({ page }) => {
  await page.waitForNavigation();
});

[yury-s]

@sb-michal-murray if it doesn't work can you share your config?

[sb-michal-murray]

Here is the config:

  {
      name: 'TestProject1',
      testDir: './testproject1/tests',
      use: {
        channel: 'chrome',
        viewport: process.env.CI ? { width: 1280, height: 720 } : { width: 1920, height: 1080 },
        ignoreHTTPSErrors: true,
        launchOptions: {
          slowMo: 50,
        },
        acceptDownloads: true,
        video: 'retain-on-failure',
      },
}

As you can see, we didn't change the default timeout, which is I think 30 seconds, but in this particular test it was set to 180 s ( by test.setTimeout() method)

[yury-s]

Here is the config:

retries are not specified in the config, so it defaults to 0.

[pavelfeldman]

Closing as per above. Test retry is opt-in! https://playwright.dev/docs/next/test-retries#retries

[sb-michal-murray]

Sorry, retries are of course specified but I only gave you the project config forgetting the rest, which is:

  retries: process.env.CI ? 2 : 0,
  reporter: process.env.CI ? [['line'], ['./utils/teamcityReporter.ts']] : 'list',
  workers: 8,