Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Multithreaded tests vs. testing utils #33146
Parallel testing feature (#31900) provides two implementations: using threads and using processes.
When dealing with threads, we should take care of thread safety. Although Rails codebase itself is a thread-safe some testing tools are not.
And the problem is not only with thread-safety but thread-isolation.
I've started to collect such potentially problematic places, and ended up with the following list:
It's not a problem when using with Minitest (i.e., include
It is pretty often used to test custom caching behavior. It's thread-safe, but all threads use the same store, which is not what we expect in multithreaded tests.
We need some test-specific cache with a per-thread store.
Not sure that the list is complete. And, of course, there are many third-party testing tools which do not care about threads.
Unfortunately, just fixing this non-thread-aware tools is not enough: we have system tests which could rely on this non-threaded functionality (e.g., sending emails in a server thread and verifying them in the main thread, time-traveling features, etc.). That's the same problem as with the shared ActiveRecord connection.
That means we need a way to toggle threaded behavior for these tools (and, btw, third-party testing utils too), probably, through a specific hook:
class ActiveSupport::TestCase parallelize(workers: 4, with: :threads) multithreaded_setup do ActionMailer::Base.delivery_method = :threaded_test end multithreaded_teardown do ActionMailer::Base.delivery_method = :test end end
Or vice versa – to add a system tests hook (IMO, a better option):
system_tests_setup do ActionMailer::Base.delivery_method = :shared_test end
I'd like to help with fixing these problems, but first, we should agree on the way to do that.
Isn't this requirement generally addressed by the
I'd be reluctant to add a system tests hook for this, because that feels like it permanently gives up ground I'm not yet ready to yield: they're incompatible with threads at the moment, but not necessarily forever. Defining "running system tests" = "using process isolation" would make that difficult to change in future.
If we default to the thread-aware implementations then system tests would break, don't they? Or I didn't understand you correctly.
Agree. It's too early to give on multithreaded system tests.
As I see, we have three different situations:
The main question is how to properly handle the third case.
Can we run a Capybara server within the same thread as the test itself? Don't think so.
Thus we need some sharable isolated environment. That's possible but, IMO, too specific and complicated.
Anyway, we have plenty of time to figure out the better way.
I guess my concern is that the split between 2 & 3 isn't so clean in practice: system tests definitely have multiple threads involved out of the box, but it's not super-improbable for other tests to have threads in flight either.
My current least-bad thought is something that involves hooking
We can also monkey-patch
Thread.prepend(Module.new do def (key) group[key] || super end end)
Then we only have to initialize the corresponding group variable in
This issue has been automatically marked as stale because it has not been commented on for at least three months.