Allow configuring the quarantine mode

[arijo]

What is your Test Scenario?

Some test are flaky in CI and we would like to increase the number of retries when running in quarantine mode.

What are you suggesting?

A way to configure the number of retries when running the tests in quarantine mode: -q n

What alternatives have you considered?

Disabling the tests and trying to make the less flaky - but flakiness can depend on other factors like cpu / memory that are not possible to change.

Additional context

Would be nice to know the number of successful tries but not really a priority.

[aleks-pro]

Hello @arijo ,

Thank you for sharing your idea with us. We have decided to add this option in future Testcafe releases. In the meantime, you are welcome to submit your PR.

[jiteshsojitra]

This is very important because default retry for -q is 3, which may be high for someone and takes longer to mark test as failed. Ideal retry for test is one more time for majority of the people. Can we prioritize this issue?

[AndreyBelym]

From our testing experiences, running a failed test just one more time doesn't reduce flakiness well. Current settings for quarantine mode feels balanced enough between robustness and time requirements. So we will be happy to accept a PR about this feature, but our team is busy with more important tasks now.

[dlangerenken]

What I think would be helpful is to have an option to run the failed test again later and not directly after the first initial fail - given that network hiccups usually resolve itself in a few seconds/minutes.

Also, if my test succeeds only once in five runs, I'm still happy enough and would like to proceed with the next tests... But I guess that's something not many teams would accept as "stable"

[aleks-pro]

Hello @dlangerenken ,

Thank you for sharing your idea with us. We will take it into account.

[Farfurix]

For the team: https://stackoverflow.com/questions/60700277/how-to-configure-testcafe-quarantine-mode-to-pass-if-there-is-at-least-1-success

[rodrigotolledo]

From my own experience, the most useful behaviour would be:

If a test succeeded in Quarantine Mode at least once, we should mark it as a PASS (even if majority of executions failed).

The main thinking behind this: If a test is in Quarantine Mode, it's because it's flaky, so it's not very realistic to expect that it will Pass majority of times. If it passes at least once, that's enough to conclude the functionality is working.

Yesterday I was having a quick look at TestCafe code and looks like this is the area that may need to be changed:

_shouldKeepInQuarantine () {
        const errors         = this.testRun.errs;
        const hasErrors      = !!errors.length;
        const attempts       = this.quarantine.attempts;
        const isFirstAttempt = this._isFirstQuarantineAttempt();

        attempts.push(errors);

        // Code below is just a draft/idea to finish Quarantine if there is at least 1 successful run. Not a final solution.
        if (errors.length == 0 && hasErrors == false && attempts > 0 && isFirstAttempt == false) {
            await this._endQuarantine();
        }

        return isFirstAttempt ? hasErrors : !this.quarantine.isThresholdReached();
    }

[rodrigotolledo]

Hello @Farfurix,

Just wondering if this work will be prioritised at some point?

Thank you!

[VadimKrim]

Same question here

[AlexSkorkin]

We didn't plan to address this issue in the nearest sprints. In the meantime, you are welcome to submit your PR with this functionality.

[rob4629]

Hey 👋🏻 . Some of our tests rely on 3rd party tools, which we have no control over (PayPal being the main one). As such, our tests sometimes run as many as 5 times to hit either 3 passes or 3 fails. This tends to cause CircleCi to timeout if the test runs take longer than 10mins to write back to the console.

I would find it really beneficial to configure Quarantine mode, to break after the first pass. OR... to have TestCafe write back to the console each time a test ends. Preferably the former though.

Alternatively, is there a way I can implement something using the TestCafe Runner?

[SergeyShurygin]

Hi @rob4629,
Thank you for sharing such use case.

Since CI terminates a task if there is no output for 10 minutes, you can add some console output at the beginning of your test. You can do this either before each hook or in the custom reporter of the reportTestStart method (https://devexpress.github.io/testcafe/documentation/guides/extend-testcafe/reporter-plugin.html#implement-the-reporter)

[rob4629]

Hey, I spent a little time looking at this, and have a PR. Set quarantineMode to be either Boolean or Dictionary: accept retryCount and passCount arguments:

testcafe chrome -q
testcafe chrome -q retryCount=2,passCount=1

Update the Threshold values based on this (if provided), otherwise use default (3).

Any feedback would be appreciated