Feature: Run until test fails (debug flakey/intermittent errors)

[MadLittleMods]

Add a way to keep re-running the test-suite until you run into a failure. When debugging flakey, intermittent failures, it would be nice to just keep retrying the tests until something fails. I don't want to keep pressing re-run manually.

This could be accomplished with cypress run and some outside scripting on top of it but I want to do this inside the cypress open test runner. Using cypress run is also tough to get useful info out of when a failure happens but this is being tracked by #448

There is a Run all tests button in the UI but it only re-runs the test suite once. I want to keep retrying the test suite until some test fails.

---

Related issues but mainly around retrying failed tests,

• Option to abort on first failure #518
• Feature: Stop Cypress when there are too many failed tests #1599
• Allow to rerun only failed tests directly from the Test Runner UI #4886

[gabbersepp]

Maybe you can take the great retry plugin and inverse its behaviour? Should not be too much efford i think

[gabbersepp]

Or open an issue in its repository. Maybe the owner introduces a flag to switch between both modes?

[MadLittleMods]

Thanks @gabbersepp! Created an issue over there, kuceb/cypress-plugin-retries#37

[jennifer-shehane]

There is an example in our recipes that kind of does something similar to this here: https://github.com/cypress-io/cypress-example-recipes/blob/master/test-repeat.js

[alxndr]

I've defined a shell function which re-runs the command given until it fails...

until_fail () {
        eval "$@"
        while [ "$?" -eq "0" ]
        do
                eval "$@"
        done
}

Example usage:

$ until_fail npm run -s cypress:run -- --spec cypress/integration/spec_file.js

[jennifer-shehane]

Please try our 'experimentalRetries’ feature. We’ll be coming out with more around flake detection soon. I’ll close this since we’ve already prioritized this work.

[ConorFayleTCO]

hi @jennifer-shehane am I right in thinking this feature is locked behind the Team plan, even for running locally? If so, that would be quite disappointing.

[gabbersepp]

hi @jennifer-shehane am I right in thinking this feature is locked behind the Team plan, even for running locally? If so, that would be quite disappointing.

We are using native retry function since 5.0.0 or so without team plan. This is the original issue where the feature was introduced:
#1313 (comment)

not sure what are the differences to the new experimental retry feature