Some way to randomize scenario (or at least feature) order?

[yarmiganosca]

A way to run scenarios in a random order would be very helpful. I'm pretty sure scenarios can be run in a random order, but if they can't, then I'd at least like the ability to run features in a random order.

If y'all aren't opposed to the idea, I can get started on an implementation. In terms of UX, I was planning on shamelessly copying RSpec's seed feature. Here's the rough outline:

• Configuration file parameters for both randomization (which will be off by default), and specific numeric seed.
• Command line flag for turning on randomization without providing a specific seed (--random)
• Command line flag for providing a specific numeric randomization seed (--seed <SEED>)
• When randomized, Spinach runs would output the randomization seed.

[mrcasals]

Looks good to me!

[yarmiganosca]

Actually starting to work on this today (just keeping you updated). My plan is to break implementing this up into 2 steps:

Step 1) Add full functionality for handling randomization & seeds, but don't implement run ordering:

• configuration options for randomization & for seeds
• something similar to RSpec's "ordering strategy" concept for deciding how a run should be ordered
• ability to receive seeds on the command line
• seeds present in test output

Again, none of these things will actually touch test order, just make sure that everything required to randomly order tests is present (you can guess what step 2 is at this point).

Step 2) Write an ordering strategy that randomize feature & scenario order.

[mrcasals]

Sounds good! @ywen any thoughts on this? :)

[ywen]

Sounds good to me! Thanks for doing this @yarmiganosca

[yarmiganosca]

@ywen No problem! We've had several test issues where the non-randomization has hidden poor setup/teardown on our part, so why not?

Also, Step 1 is done, and relatively non-invasively. I only had to change the assumptions in one of the existing tests, and not very much. Here's a link if you want to look at it: yarmiganosca/spinach@master...yarmiganosca:19fa42299c10943b86e7d833b23ee145a7ede683

For Step 2, I don't think I can remain this non-invasive. The current way Runner handles the array of files, and the way FeatureRunner sometimes only runs scenarios make a clean insertion of randomization almost impossible. How opposed would y'all be if I did some refactoring along the way that made my task easier? Nothing aggressive or outside the scope of the task, just what I needed to do this.

[mrcasals]

👍 for me!

[ywen]

@yarmiganosca I wouldn't mind refactoring. It is welcomed actually. :)

[yarmiganosca]

Good to hear! I'll probably get started on that next week.

[yarmiganosca]

I don't mean to be a pest (especially since I took a year to finish the feature), but any chance we can get a release of this? Thanks!

[yarmiganosca]

Nvm. Just realized I can send you a PR for the version bump. 🤦‍♂️ Still getting used to this whole "contributing to open source" thing.