Skip to content

Add 'test_repeatedly', for help with flakey tests#85

Merged
joshpencheon merged 2 commits intomasterfrom
test-repeatedly
Jan 28, 2021
Merged

Add 'test_repeatedly', for help with flakey tests#85
joshpencheon merged 2 commits intomasterfrom
test-repeatedly

Conversation

@joshpencheon
Copy link
Contributor

Issue Summary

Often, when investigating whether a test is flakey, it's helpful to be able to run a test multiple times. This tends to get done by either using a bash loop (which can be slow) or hacking about in the test file (which can get messy).

Using a bash loop to repeatedly instantiate a runner is slow:

for i in `seq 1 100`; do bin/rails test path/to/test.rb:123; done

And mucking around duplicating the test gets messy:

+ 100.times do |x|
- test 'something flakey' do
-   # ...
- end
+   test "something flakey - #{x}" do
+     # ...
+   end
+ end

This PR

This PR adds the test_repeatedly construct, to make the latter pattern (which executes more efficiently) quick and simple to do, whilst retaining some configurability.

- test 'something flakey' do
+ test_repeatedly 'something flakey' do
    # 
  end

Copy link
Contributor

@ollietulloch ollietulloch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @joshpencheon - I've pointed an app at this branch locally and oberved an integration test running repeatedly for a given number of times, or 100 times if not specified.

@joshpencheon joshpencheon merged commit a18f358 into master Jan 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants