Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
bats: add support for deteriministic --shuffle #196
There are many cases where tests accidentially depend on one another,
In order to avoid a dependency on GNU coreutils (for OS X users), we
This is a follow-up of #172.
Signed-off-by: Aleksa Sarai email@example.com
This still needs some tests -- though I'm a little cautious about how we're going to test the seed values in a way that doesn't cause us problems in the future (if
referenced this pull request
Mar 12, 2019
7 times, most recently
Mar 16, 2019
This was referenced
Apr 25, 2019
@jasonkarns As far as I'm aware there aren't many other testing tools which provide something like this. @sublimino mentioned this in #171. Ruby has
@cyphar Virtually every testing tool that I'm aware of, which supports random execution, also supports re-running a prior random order deterministically.
Minitest is random by default and accepts seed via
Mochajs doesn't officially support random order, but the "heavily used" patch uses
Tap, Jest, Ava, Cypress don't seem to support true randomization nor determinism with a seed; though they support parallelization of the suite so there's some implicit randomicity but without the ability to re-run that particular order. (And they can both force serial runs, which of course isn't the same as deterministically re-running a random order.)
The rest of the common JS tools (tape, teenytest, nightwatch, qunit) don't seem to support randomization at all.
I would be curious to see what features the top 2 or 3 testing tools of python, php, perl, go, java, .net support. At the very least
Personally, I do like cucumber, rspec, and mochajs' dual use of a single flag for both enabling randomization and providing the seed (since a seed without random ordering is meaningless).
I will go with
Is not a large number of tools, not that no tools which support random execution support re-using the same order (in fact I would argue the opposite because it's such an obvious thing to need).