[Feature] Add config option to not report retries

[iansan5653]

Problem

If a non-flaky test fails and retries are enabled, the reporting almost always results in very redundant output. For example, only one test failed out of 11 in this run, but the output is still almost 100 lines:

Output from dot reporter for a test suite

Running 11 tests using 8 workers
·×······×···×F

  1) [chromium] › layouts/table/table-selection-spec.ts:29:9 › navigationAndFocus › Selecting rows › shift+space selects a row and escape clears selection 

    Error: expect(received).toBe(expected) // Object.is equality

    Expected: false
    Received: true

      31 |
      32 |       const [row1] = await getRows(page, ['0'])
    > 33 |       expect(await getRowSelectionState(row1)).toBe(false)
         |                                                ^
      34 |
      35 |       await page.keyboard.press('Escape')
      36 |

        at src/playwright-tests/layouts/table/table-selection-spec.ts:33:48

    attachment #1: screenshot (image/png) ──────────────────────────────────────────────────────────
    src/playwright-tests/artifacts/layouts-table-table-selection-spec.ts-navigati-df80e-space-selects-a-row-and-escape-clears-selection-chromium/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

    Retry #1 ───────────────────────────────────────────────────────────────────────────────────────

    Error: expect(received).toBe(expected) // Object.is equality

    Expected: false
    Received: true

      31 |
      32 |       const [row1] = await getRows(page, ['0'])
    > 33 |       expect(await getRowSelectionState(row1)).toBe(false)
         |                                                ^
      34 |
      35 |       await page.keyboard.press('Escape')
      36 |

        at src/playwright-tests/layouts/table/table-selection-spec.ts:33:48

    attachment #1: video (video/webm) ──────────────────────────────────────────────────────────────
    src/playwright-tests/artifacts/layouts-table-table-selection-spec.ts-navigati-df80e-space-selects-a-row-and-escape-clears-selection-chromium-retry1/video.webm
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #2: screenshot (image/png) ──────────────────────────────────────────────────────────
    src/playwright-tests/artifacts/layouts-table-table-selection-spec.ts-navigati-df80e-space-selects-a-row-and-escape-clears-selection-chromium-retry1/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

    Retry #2 ───────────────────────────────────────────────────────────────────────────────────────

    Error: expect(received).toBe(expected) // Object.is equality

    Expected: false
    Received: true

      31 |
      32 |       const [row1] = await getRows(page, ['0'])
    > 33 |       expect(await getRowSelectionState(row1)).toBe(false)
         |                                                ^
      34 |
      35 |       await page.keyboard.press('Escape')
      36 |

        at src/playwright-tests/layouts/table/table-selection-spec.ts:33:48

    attachment #1: screenshot (image/png) ──────────────────────────────────────────────────────────
    src/playwright-tests/artifacts/layouts-table-table-selection-spec.ts-navigati-df80e-space-selects-a-row-and-escape-clears-selection-chromium-retry2/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

    Retry #3 ───────────────────────────────────────────────────────────────────────────────────────

    Error: expect(received).toBe(expected) // Object.is equality

    Expected: false
    Received: true

      31 |
      32 |       const [row1] = await getRows(page, ['0'])
    > 33 |       expect(await getRowSelectionState(row1)).toBe(false)
         |                                                ^
      34 |
      35 |       await page.keyboard.press('Escape')
      36 |

        at src/playwright-tests/layouts/table/table-selection-spec.ts:33:48

    attachment #1: screenshot (image/png) ──────────────────────────────────────────────────────────
    src/playwright-tests/artifacts/layouts-table-table-selection-spec.ts-navigati-df80e-space-selects-a-row-and-escape-clears-selection-chromium-retry3/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

  1 failed
    [chromium] › layouts/table/table-selection-spec.ts:29:9 › navigationAndFocus › Selecting rows › shift+space selects a row and escape clears selection 
  10 passed (28.8s)

As you can see, the failure output is duplicated four times (once for the initial test run and three more times because retries is set to 3).

This is also problematic with other reporters - for example, the github reporter outputs 4 annotations on a single line for this test failure:

This is very noisy and almost never has value beyond indicating that retries were attempted.

Proposed solution

I propose that we add a new retries configuration parameter to the built-in reporters. This would have three available options:

• "all" (default): print all retries, no change from current behavior
• "none": only print the initial test failure, plus a short summary of the retries (ie, "3 retry attempts also failed")
• "last": only print the last retry failure. This is useful both for indicating the number of retries that were attempted and for getting the video artifact path when video is set to 'retry-with-video'

Alternatively, we could use true, false, and "last" instead. I don't have strong opinions on this.

[iansan5653]

If this sounds like a good idea, I'd be happy to open a PR to add this feature.

[EchooWww]

Is there a PR for this feature yet 👀

[waterplea]

That would be very helpful in reducing noise.

[fspinillo]

Want to echo this would be a valuable change. The noise is very high when evaluating a log of a test run