Skip to content

Commit

Permalink
Specify that random order is stable
Browse files Browse the repository at this point in the history
In addition to predictability, another useful property of random
ordering is stability, i.e. if a given seed results in the order BCDA,
then skipping a step will not affect the relative order of the other
steps: if C is skipped, the order should be BDA. This is a necessary
precondition for bisection, should we wish to add that in the future.

As randomisation takes place before filtering we get stability for free.
This change just documents it and preserves it from future regression.
  • Loading branch information
threedaymonk committed Apr 26, 2016
1 parent 16d2277 commit eb6e54c
Showing 1 changed file with 64 additions and 6 deletions.
70 changes: 64 additions & 6 deletions features/docs/cli/randomize.feature
Expand Up @@ -9,6 +9,10 @@ Feature: Randomize
you can reproduce that run by using the seed that's printed at the end of
the test run.

For a given seed, the order of scenarios is constant, i.e. if step A runs
before step B, it will always run before step B even if other steps are
skipped.

Background:
Given a file named "features/bad_practice_part_1.feature" with:
"""
Expand All @@ -27,6 +31,14 @@ Feature: Randomize
Scenario: Depend on state from a preceding feature
When I depend on the state
"""
And a file named "features/unrelated.feature" with:
"""
Feature: Unrelated
@skipme
Scenario: Do something unrelated
When I do something
"""
And a file named "features/step_definitions/steps.rb" with:
"""
Given(/^I set some state$/) do
Expand All @@ -36,6 +48,9 @@ Feature: Randomize
Given(/^I depend on the state$/) do
raise "I expect the state to be set!" unless $global_state == "set"
end
Given(/^I do something$/) do
end
"""

Scenario: Run scenarios in order
Expand All @@ -44,7 +59,7 @@ Feature: Randomize

@spawn
Scenario: Run scenarios randomized
When I run `cucumber --order random:41515 -q`
When I run `cucumber --order random:41544 -q`
Then it should fail
And the stdout should contain exactly:
"""
Expand All @@ -56,6 +71,12 @@ Feature: Randomize
./features/step_definitions/steps.rb:6:in `/^I depend on the state$/'
features/bad_practice_part_1.feature:7:in `When I depend on the state'
Feature: Unrelated
@skipme
Scenario: Do something unrelated
When I do something
Feature: Bad practice, part 2
Scenario: Depend on state from a preceding feature
Expand All @@ -72,11 +93,48 @@ Feature: Randomize
Failing Scenarios:
cucumber features/bad_practice_part_1.feature:6
cucumber features/bad_practice_part_2.feature:3
4 scenarios (2 failed, 2 passed)
4 steps (2 failed, 2 passed)
Randomized with seed 41544
"""

@spawn
Scenario: Run scenarios randomized with some skipped
When I run `cucumber --tags ~@skipme --order random:41544 -q`
Then it should fail
And the stdout should contain exactly:
"""
Feature: Bad practice, part 1
Scenario: Depend on state from a preceding scenario
When I depend on the state
I expect the state to be set! (RuntimeError)
./features/step_definitions/steps.rb:6:in `/^I depend on the state$/'
features/bad_practice_part_1.feature:7:in `When I depend on the state'
Feature: Bad practice, part 2
Scenario: Depend on state from a preceding feature
When I depend on the state
I expect the state to be set! (RuntimeError)
./features/step_definitions/steps.rb:6:in `/^I depend on the state$/'
features/bad_practice_part_2.feature:4:in `When I depend on the state'
Feature: Bad practice, part 1
Scenario: Set state
Given I set some state
Failing Scenarios:
cucumber features/bad_practice_part_1.feature:6
cucumber features/bad_practice_part_2.feature:3
3 scenarios (2 failed, 1 passed)
3 steps (2 failed, 1 passed)
Randomized with seed 41515
"""
Randomized with seed 41544
"""

0 comments on commit eb6e54c

Please sign in to comment.