Retrying Flaky Tests

Retrying Flaky Tests

A flaky test fails, then passes on a re-run without anything changing. Plum can retry failed scenarios automatically and marks the ones that recovered so they don't hide in a green report.

The setting

Settings → Project → Max retries (owner, or an admin on the project). 0 disables it; the maximum is 5. It applies to every automated run in that project — the run bar, the CLI trigger, schedules and MCP.

How it works, per framework

The rest of the run — browser, timeouts, traces, reporters — comes from the project's config. Retries are the exception:

• Playwright reports every attempt in its JSON, so Plum passes --retries=N straight through and lets Playwright do the retrying.
• Cucumber's legacy JSON reports only the final attempt, so Plum re-runs the failures itself: after a pass it scopes the next attempt to just the still-failing scenarios and folds the results together.

Either way the report ends up with the same shape. When a run is split across nodes, each node applies the same retry count to its own slice.

In the report

• A scenario that failed at least once, then passed is marked flaky.
• A scenario that never passed is just failed, not flaky.
• The reports list flags a run as flaky (a small badge) when it contains any flaky scenario, without loading the full report — so a "100% passing" run still tells you something recovered on retry.

Reading the signal

A flaky badge means: the test is real but unreliable. Common causes are timing (a missing await for a network state), test-order coupling, or a genuinely slow environment. Open the session replay for the failed attempt to see the state the browser was actually in.