You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In order to improve the quality of user tests, it would be great to run tests in a random order based on a random seed to ensure that the tests are properly isolated from each other.
The use of a seed will allow to re-run tests in the same order to reproduce a failed case for a specific order.
Other test suites have this feature; for example, you can refer to jest randomize or vitest shuffle
The text was updated successfully, but these errors were encountered:
Do you mean the suite files, or the tests within a given file?
Tap doesn't defer your tests or put them in any sort of queue to be run at some later time, so "shuffling" is sort of a weird thing to consider. Tap just runs your test functions right away, unless it's already in the middle of something else.
a
TAP version 14
in b
# Subtest: b
1..0
ok 1 - b # time=0.781ms
c
in d
# Subtest: d
1..0
ok 2 - d # time=0.336ms
e
1..2
# { total: 2, pass: 2 }
# time=9.289ms
As you can see, it doesn't even know about the d subtest until the b subtest is completely finished and resolved. This determinism is baked into tap, so that definitely could not be changed by default, maybe as opt-in, and only if the tests are not sync.
If they're async (and maybe whatever shuffle opt-in forces them to be deferred and async), they could theoretically be shuffled. Today, if you set t.jobs to a number > 1, then it will run them in parallel, which shuffles them up a bit, but not quite what you're looking for, I think.
It's really not required for the tests within a given file to be isolated from one another, and it's pretty common for them not to be. For example, you might be testing specifically that a given series of actions has some effect, and using subtests as a sort of flow-control-plus-assertions is a handy way to do that.
Approved feature: --shuffle flag will run test files in a random order, and print the seed that was used, which can also be set via --shuffle-seed=<number>. It won't have any effect on the contents of each individual test file, though.
In order to improve the quality of user tests, it would be great to run tests in a random order based on a random seed to ensure that the tests are properly isolated from each other.
The use of a seed will allow to re-run tests in the same order to reproduce a failed case for a specific order.
Other test suites have this feature; for example, you can refer to jest randomize or vitest shuffle
The text was updated successfully, but these errors were encountered: