Command-line flag to run specs multiple times

[afeld]

I have some integration tests that are failing inconsistently, and as I'm fixing them I'd love to be able to run specific specs multiple times to see which still fail intermittently. Something like the following:

$ rspec -n 3 spec
...
  should pass consistently

Finished in 43 seconds
1 example, 1/3 failures

Thoughts?

[afeld]

I'm currently just using a loop in bash to do this:

for i in `seq 10`; do rspec spec; done

but with a lot of puts from the server, it's hard to pull out the test results for each run from the noise.

[alindeman]

What specific advantages would rspec -n have over the for loop?

[afeld]

My rspec output is separated by a gajillion lines of logging, enough to push the output of previous runs out of the shell history. I suppose when using for I could

• redirect output from rspec (separate from app output, if that's possible?), or
• redirect all output to a file and then grep it for the results

...though seems like this issue might come up for enough people where -n (or equivalent) would be a generally useful flag. Where --order rand gives some confidence of idempotence between individual specs, this would do the same for multiple runs of the same test.

[dchelimsky]

@afeld what sorts of intermittent failures are revealing themselves when you run examples in a loop?

[afeld]

We have single-pagey app, so with specs that drive selenium, there are sometime race conditions where the test is checking for the presence of an element on the page that gets created via JS (and sometimes needs to wait on AJAX). We're using capybara, but it's a bit opaque as to when it waits for things to complete and when it doesn't.

Is there a way to capture the rspec output specifically (in the shell or programmatically)? I'm happy to create a wrapper gem for this if it doesn't seem appropriate in core. I shall call it am_i_nuts_or_did_i_just_see_that_spec_fail.

cc: @jnicklas

[dchelimsky]

Is there a way to capture the rspec output specifically (in the shell or programmatically)? I'm happy to create a wrapper gem for this if it doesn't seem appropriate in core.

I'd say a wrapper gem makes sense. We used to do this in rspec-rails, but I don't think it does anymore. Cucumber probably does so you can peek at that for ideas.

[afeld]

Will do. If anyone else knows of other examples that hook into rspec to retrieve the results, please let me know!

[afeld]

Apparently I wasn't the first person to think of this: https://github.com/oggy/respec, https://github.com/dblock/rspec-rerun and #456 all refer to re-running failed examples... I'll probably do a pull request on one of them to have an option to run X times and report which tests are flaky.

[taylor]

Try this for seeing when things fail:

for i in `seq 50` ; do rspec spec ; [[ ! $? = 0 ]] && break ; done

[avit]

Running the test suite repeatedly (in the same process) would also be helpful for tracing memory leaks.

[lionelperrin]

Hello,

I discover this request right now, looking for the same option.

This ability to repeat a test suite exists in several test framework. For instance https://code.google.com/p/googletest/wiki/AdvancedGuide#Repeating_the_Tests.

This is useful to detect memory leak and in general checking that resources are correctly managed. (File closed as expected, no shadow child process or thread, etc...) Spawning several time using a bash script is not suitable in these case because cleanup will be performed by the OS.

In addition, as stated in this issue, the output of the bash script is not very user friendly.

Regards,

Lionel

[avit]

@lionelperrin the linked issue #1852 that duplicates this one also suggests running repeated tests out-of-process. I agree, this doesn't address the use case of testing for object allocations/memory leaks, or for repeatability between test suite setup & teardown.

[JonRowe]

If you want to check for object allocations / memory leaks you're better off writing a test that checks for that specifically by instantiating and measuring within that spec, that way you eliminate the extra noise that RSpec itself introduces into that environment.