Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Reuse test workers #302
So far, Criterion spawns a whole new process per test. This makes things simple implementation-wise, but it's otherwise not that great on a performance standpoint. execve()s are pretty fast these days, but they still have a non-negligible cost.
An obvious solution here would be to simply have a pool of boxfort workers that each run a bunch of tests sequentially. If a worker dies or exits, we can still tie the failure to the currently executing test, and respawn the worker for more processing.
Yes, because just fork()ing had some problems.
The first issue was that the network part of criterion was not fork-safe, and trying to make nanomsg fork-safe was pretty-hard. I had to maintain a set of patches to allow that, and even then it was a bit unstable.
The other issue were that anything that the runner was doing in its initialization leaked to the worker process, which caused further issues with standard tools like valgrind or the compiler sanitizers.
That's a good point. I'm willing to bet that this is not something most tests care about though, so I'd probably be on the side of adding an option to preserve the single-process isolation.
I'm more worried about memory corruption creeping from another test though, but compiling them with address-sanitizer could be a reasonable compromise?