From eb6e54c3fde67f6f8b031907857b2eb1e06015dc Mon Sep 17 00:00:00 2001
From: Paul Battley <pbattley@gmail.com>
Date: Tue, 26 Apr 2016 15:18:37 +0100
Subject: [PATCH] Specify that random order is stable

In addition to predictability, another useful property of random
ordering is stability, i.e. if a given seed results in the order BCDA,
then skipping a step will not affect the relative order of the other
steps: if C is skipped, the order should be BDA. This is a necessary
precondition for bisection, should we wish to add that in the future.

As randomisation takes place before filtering we get stability for free.
This change just documents it and preserves it from future regression.
---
 features/docs/cli/randomize.feature | 70 ++++++++++++++++++++++++++---
 1 file changed, 64 insertions(+), 6 deletions(-)

diff --git a/features/docs/cli/randomize.feature b/features/docs/cli/randomize.feature
index d3ff2ad42e..93012d442e 100644
--- a/features/docs/cli/randomize.feature
+++ b/features/docs/cli/randomize.feature
@@ -9,6 +9,10 @@ Feature: Randomize
   you can reproduce that run by using the seed that's printed at the end of
   the test run.
 
+  For a given seed, the order of scenarios is constant, i.e. if step A runs
+  before step B, it will always run before step B even if other steps are
+  skipped.
+
   Background:
     Given a file named "features/bad_practice_part_1.feature" with:
       """
@@ -27,6 +31,14 @@ Feature: Randomize
         Scenario: Depend on state from a preceding feature
           When I depend on the state
       """
+    And a file named "features/unrelated.feature" with:
+      """
+      Feature: Unrelated
+
+        @skipme
+        Scenario: Do something unrelated
+          When I do something
+      """
     And a file named "features/step_definitions/steps.rb" with:
       """
       Given(/^I set some state$/) do
@@ -36,6 +48,9 @@ Feature: Randomize
       Given(/^I depend on the state$/) do
         raise "I expect the state to be set!" unless $global_state == "set"
       end
+
+      Given(/^I do something$/) do
+      end
       """
 
   Scenario: Run scenarios in order
@@ -44,7 +59,7 @@ Feature: Randomize
 
   @spawn
   Scenario: Run scenarios randomized
-    When I run `cucumber --order random:41515 -q`
+    When I run `cucumber --order random:41544 -q`
     Then it should fail
     And the stdout should contain exactly:
       """
@@ -56,6 +71,12 @@ Feature: Randomize
             ./features/step_definitions/steps.rb:6:in `/^I depend on the state$/'
             features/bad_practice_part_1.feature:7:in `When I depend on the state'
 
+      Feature: Unrelated
+
+        @skipme
+        Scenario: Do something unrelated
+          When I do something
+
       Feature: Bad practice, part 2
 
         Scenario: Depend on state from a preceding feature
@@ -72,11 +93,48 @@ Feature: Randomize
       Failing Scenarios:
       cucumber features/bad_practice_part_1.feature:6
       cucumber features/bad_practice_part_2.feature:3
-    
+
+      4 scenarios (2 failed, 2 passed)
+      4 steps (2 failed, 2 passed)
+
+      Randomized with seed 41544
+
+      """
+
+  @spawn
+  Scenario: Run scenarios randomized with some skipped
+    When I run `cucumber --tags ~@skipme --order random:41544 -q`
+    Then it should fail
+    And the stdout should contain exactly:
+      """
+      Feature: Bad practice, part 1
+
+        Scenario: Depend on state from a preceding scenario
+          When I depend on the state
+            I expect the state to be set! (RuntimeError)
+            ./features/step_definitions/steps.rb:6:in `/^I depend on the state$/'
+            features/bad_practice_part_1.feature:7:in `When I depend on the state'
+
+      Feature: Bad practice, part 2
+
+        Scenario: Depend on state from a preceding feature
+          When I depend on the state
+            I expect the state to be set! (RuntimeError)
+            ./features/step_definitions/steps.rb:6:in `/^I depend on the state$/'
+            features/bad_practice_part_2.feature:4:in `When I depend on the state'
+
+      Feature: Bad practice, part 1
+
+        Scenario: Set state
+          Given I set some state
+
+      Failing Scenarios:
+      cucumber features/bad_practice_part_1.feature:6
+      cucumber features/bad_practice_part_2.feature:3
+
       3 scenarios (2 failed, 1 passed)
       3 steps (2 failed, 1 passed)
-      
-      Randomized with seed 41515
-      
-      """
 
+      Randomized with seed 41544
+
+      """