Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bats: add support for deteriministic --shuffle #196

Open
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
2 participants
@cyphar
Copy link
Contributor

commented Mar 5, 2019

There are many cases where tests accidentially depend on one another,
and bats users might want to ensure their test suite does not have this
behaviour. While --jobs gives you some randomness in the execution,
--shuffle allows for deterministic ordering in order to re-test a
particular case (by setting BATS_SHUFFLE_SEED) and also shuffles all
tests entirely.

In order to avoid a dependency on GNU coreutils (for OS X users), we
can't use shuf(1) and instead have to resort to a fairly contorted usage
of paste(1). On the plus side is means we can be sure that the ordering
won't change with new shuf(1) versions.

This is a follow-up of #172.

Signed-off-by: Aleksa Sarai cyphar@cyphar.com

@cyphar cyphar requested a review from bats-core/bats-core as a code owner Mar 5, 2019

@cyphar

This comment has been minimized.

Copy link
Contributor Author

commented Mar 5, 2019

This still needs some tests -- though I'm a little cautious about how we're going to test the seed values in a way that doesn't cause us problems in the future (if awk's srand starts producing different values). I will work on that when I have some time.

@cyphar cyphar referenced this pull request Mar 12, 2019

Merged

bats: support parallel execution of tests #172

2 of 2 tasks complete

@cyphar cyphar force-pushed the cyphar:bats-shuffle branch 7 times, most recently from 648e3ac to 0ad3d99 Mar 16, 2019

@cyphar cyphar force-pushed the cyphar:bats-shuffle branch 2 times, most recently from d09dd86 to 03b17b2 Apr 3, 2019

@cyphar

This comment has been minimized.

Copy link
Contributor Author

commented Apr 8, 2019

As an update, it looks like old bash versions don't like the arrays_equal code. I will need to take a look at this...

@jasonkarns

This comment has been minimized.

Copy link
Member

commented Apr 8, 2019

I'm 👍 on the feature. Haven't looked at the implementation, though. But before we settle too hard on the interface (the flag and var names), can we put together a quick survey of what other tools are using for their CLI flags/options/var names? That way we can follow any existing norms, however, slight they may be?

bats: add support for deteriministic --shuffle
There are many cases where tests accidentially depend on one another,
and bats users might want to ensure their test suite does not have this
behaviour. While --jobs gives you some randomness in the execution,
--shuffle allows for deterministic ordering in order to re-test a
particular case (by setting BATS_SHUFFLE_SEED) and also shuffles all
tests entirely.

In order to avoid a dependency on GNU coreutils (for OS X users), we
can't use shuf(1) and instead have to resort to a fairly contorted usage
of paste(1). On the plus side is means we can be sure that the ordering
won't change with new shuf(1) versions.

Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>

@cyphar cyphar force-pushed the cyphar:bats-shuffle branch from 03b17b2 to db06e1f May 1, 2019

@cyphar

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2019

@jasonkarns As far as I'm aware there aren't many other testing tools which provide something like this. @sublimino mentioned this in #171. Ruby has --seed and random ordering is the default, so I could add another command-line flag if you like (but the default ordering is going to be in-order to avoid breaking people's tests).

@jasonkarns

This comment has been minimized.

Copy link
Member

commented May 1, 2019

@cyphar Virtually every testing tool that I'm aware of, which supports random execution, also supports re-running a prior random order deterministically.

RSpec:

--order defined
--order rand
--order rand:123
--seed 123 # same as --order rand:123

Minitest is random by default and accepts seed via -s, --seed or SEED env var.
(with an in-test helper to force non-random order: i_suck_and_my_tests_are_order_dependent!)

Cucumber uses --order random and --order random:SEED

In JavaScript/node land:

Mochajs doesn't officially support random order, but the "heavily used" patch uses --random SEED.

Jasmine has --random=true|false and --seed=SEED

Tap, Jest, Ava, Cypress don't seem to support true randomization nor determinism with a seed; though they support parallelization of the suite so there's some implicit randomicity but without the ability to re-run that particular order. (And they can both force serial runs, which of course isn't the same as deterministically re-running a random order.)

The rest of the common JS tools (tape, teenytest, nightwatch, qunit) don't seem to support randomization at all.

I would be curious to see what features the top 2 or 3 testing tools of python, php, perl, go, java, .net support. At the very least --seed so far seems to be a de facto standard for the seed.

@jasonkarns

This comment has been minimized.

Copy link
Member

commented May 1, 2019

Personally, I do like cucumber, rspec, and mochajs' dual use of a single flag for both enabling randomization and providing the seed (since a seed without random ordering is meaningless).

--order rand[:seed] is a bit heavy on the parsing. --random [SEED] is a simple compromise (the seed argument is optional).

--seed SEED on its own implying randomization seems to be the most straightforward of all the options. It follows the "convention" that all of these tools use for --seed, and since --seed without randomization is meaningless, it makes sense to me that the presence of the flag also triggers the shuffling. The downside, is that we would still need a separate flag to trigger shuffling without a seed.

@cyphar

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2019

I will go with --random [SEED] then. My point was that:

Virtually every testing tool that I'm aware of, which supports random execution, also supports re-running a prior random order deterministically.

Is not a large number of tools, not that no tools which support random execution support re-using the same order (in fact I would argue the opposite because it's such an obvious thing to need).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.