Possibility of multithreading?

[jgaskins]

With 2 of the big 3 Ruby implementations supporting a GIL-free threading model, I've been playing around with the idea of adding an option to separate spec examples across different threads for improved spec runtime.

Not everything about this will be perfect, which is why it should be an option and not enforced.

Won't reduce initial load time

AFAIK, requiring dependencies for the code under test should be sequential. I don't think requires are thread-safe.

Creating threads isn't free

Paying the cost of creating threads in very small apps/gems with short sequential spec runtimes may increase runtimes slightly. Additionally, Rubies with a GIL would pay this penalty without the ability to run specs concurrently at all.

Specs that depend on state outside of its scope get wrecked

A lot of Rails specs use something like config.before(:each) { DatabaseCleaner.clean }. This will wreck any specs that hit the DB. The good news is that spreading out specs across threads encourages developers to write specs to be independent of outside state.

However, this could lead to false passes, as well, which can be even more problematic. The best thing here might just be to encourage developers who aren't confident that their specs are 100% independent of each other to refrain from using this feature.

But there is good here

The pro, obviously, is greatly reduced spec runtime when using JRuby or Rubinius on multi-core machines. This works out much better for larger apps with hundreds of specs. This may also encourage development on these particular Rubies.

Proposal

One route I thought about was providing an option to describe and context blocks, but that would get repetitive in the spec code and would be specific to the block, disregarding the Ruby implementation on which it's being run. For CI running against multiple Rubies, this could extend MRI runtimes.

If we provide a CLI option (such as -j used by make), developers can choose whether or not to use the feature. I'm not sure how this could be configured to be specific to the Ruby implementation for CI, but it seems like it'd be easier.

The idea was to use a thread pool and a queue — either queue up the worker threads, pop them off from the main thread and pass spec examples to them or queue up the examples and let the threads pop them off and execute them. I'll be honest, I haven't had a chance to look at the code in depth yet so I don't know whether either of these would be the best solution (or even a viable solution) or whether another might be better. I'd definitely like to work on this a bit once I've had a chance to familiarize myself with the code, but I wanted to see what y'all thought about it first.

[myronmarston]

There are already a number of projects that help run specs in parallel. There's certainly room to improve, but I'm of the opinion that this belongs in an external gem, not in RSpec itself. However, if there are extension points that would make this easier that RSpec is not exposing, we can certainly discuss adding those.

[justinko]

I'm waiting for someone to integrate RSpec with Celluloid

As @myronmarston said, let us know if you need extension points.

Closing.

[jgaskins]

All good. Celluloid was actually one of the things I was thinking about, but I wouldn't have wanted to add it as a dependency directly into RSpec. It's probably a bit more than what we'd need for this.

@myronmarston Could you expand on why you feel this belongs outside of RSpec?

[myronmarston]

It's not core to RSpec. It's important that we keep rspec-core very light weight, both in terms of preventing bloat and also because it would be hard for the core team to maintain rspec-core if we're continually adding new features that can be implemented as decoupled extensions. Also, I can imagine a well-written parallel test runner being useful for more than just RSpec--so making it a separate project makes it easy for it to have an rspec adapter, a minitest adapter, etc. Lastly, this isn't just a small syntax change or new DSL method...it's an entirely different way of running the specs that (to me, at least) feels like it belongs in its own self-contained gem.

[Bill]

+1 for -j option. We are working on an integrated test that hits S3 in a before(:each). Rather than bundling multiple examples into one to reduce the number of before-block invocations it'd be great to just say rspec -j10 and spin up 10 threads in parallel. The S3 stuff consumes no CPU: it's just blocked on network latency.

[rkh]

-j option would be great. None of the above solutions just does threads.