.disableConcurrency flag not respected in quarantine mode

[Makavelic]

What is your Scenario?

I have a fixture with .disableConcurrency set and am running tests with -c 3 with quarantine mode on. I expect each test to run sequentially even when the test fails (to ensure each test cleans up before another starts, the actual order in which they run doesn't matter).

What is the Current behavior?

Quarantine mode ignores .disableConcurrency

What is the Expected behavior?

Quarantine mode should still respect .disableConcurrency and only one test at a time should execute.

What is the public URL of the test page? (attach your complete example)

Website not needed to demonstrate issue.

What is your TestCafe test code?

let someValue = false;

fixture("quarantineTest")
.beforeEach(async (t) => {
await t.wait(1);
someValue = false;
})
.afterEach(async (t) => {
await t.wait(1);
someValue = true;
}).disableConcurrency;

test("test1", async (t) => {
await t.wait(1000);
await t.expect(someValue).notOk("Verify value reset");
throw new Error();
});

test("test2", async (t) => {
await t.wait(1000);
await t.expect(someValue).notOk("Verify value reset");
throw new Error();
});

test("test3", async (t) => {
await t.wait(1000);
await t.expect(someValue).notOk("Verify value reset");
});

test("test4", async (t) => {
await t.wait(1000);
await t.expect(someValue).notOk("Verify value reset");
});

test("test5", async (t) => {
await t.wait(1000);
await t.expect(someValue).notOk("Verify value reset");
});

test("test6", async (t) => {
await t.wait(1000);
await t.expect(someValue).notOk("Verify value reset");
});

test("test7", async (t) => {
await t.wait(1000);
await t.expect(someValue).notOk("Verify value reset");
});

test("test8", async (t) => {
await t.wait(1000);
await t.expect(someValue).notOk("Verify value reset");
});

Your complete configuration file

Not needed

Your complete test report

See screenshots

Screenshots

Screenshot when c = 3

Screenshot when c = 1

Steps to Reproduce

Run fixture with cli:

node testcafe 'chrome:headless' 'pathToFile\quarantineTest.ts' '-c' '3' '-q' 'successThreshold=1,attemptLimit=3' '--fixture' 'quarantineTest'

Notice how tests 3-8 can be unstable

Run again with -c =1
node testcafe 'chrome:headless' 'pathToFile\quarantineTest.ts' '-c' '1' '-q' 'successThreshold=1,attemptLimit=3' '--fixture' 'quarantineTest'

Notice how there are no unstable tests. This demonstreates that quarantineMode ignores .disableConcurrency

TestCafe version

3.4.0

Node.js version

18.16.0

Command-line arguments

node testcafe 'chrome:headless' 'pathToFile\quarantineTest.ts' '-c' '3' '-q' 'successThreshold=1,attemptLimit=3' '--fixture' 'quarantineTest'

Browser name(s) and version(s)

Browser doesn't matter

Platform(s) and version(s)

Windows 11

Other

No response

[github-actions]

We appreciate you taking the time to share information about this issue. We reproduced the bug and added this ticket to our internal task queue. We'll update this thread once we have news.

[github-actions]

Release v3.5.0-rc.1 addresses this.

[github-actions]

Release v3.5.0-rc.1 addresses this.