Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reuse test workers #302

Open
Snaipe opened this issue Sep 13, 2019 · 4 comments

Comments

@Snaipe
Copy link
Owner

commented Sep 13, 2019

So far, Criterion spawns a whole new process per test. This makes things simple implementation-wise, but it's otherwise not that great on a performance standpoint. execve()s are pretty fast these days, but they still have a non-negligible cost.

An obvious solution here would be to simply have a pool of boxfort workers that each run a bunch of tests sequentially. If a worker dies or exits, we can still tie the failure to the currently executing test, and respawn the worker for more processing.

The -j option would then control the size of the worker pool.

@Snaipe Snaipe self-assigned this Sep 13, 2019
@Snaipe Snaipe added this to TODO in Criterion 3.0 Sep 13, 2019
@kugel-

This comment has been minimized.

Copy link

commented Sep 13, 2019

How will you ensure that each tests starts with a clean, reproducible environment?

Some of our tests play games with signal handlers and symbol interposition.

@kugel-

This comment has been minimized.

Copy link

commented Sep 13, 2019

Also, do you really do execve()? I thought criterion only fork()s

@Snaipe

This comment has been minimized.

Copy link
Owner Author

commented Sep 13, 2019

Also, do you really do execve()? I thought criterion only fork()s

Yes, because just fork()ing had some problems.

The first issue was that the network part of criterion was not fork-safe, and trying to make nanomsg fork-safe was pretty-hard. I had to maintain a set of patches to allow that, and even then it was a bit unstable.

The other issue were that anything that the runner was doing in its initialization leaked to the worker process, which caused further issues with standard tools like valgrind or the compiler sanitizers.

@Snaipe

This comment has been minimized.

Copy link
Owner Author

commented Sep 13, 2019

How will you ensure that each tests starts with a clean, reproducible environment? Some of our tests play games with signal handlers and symbol interposition.

That's a good point. I'm willing to bet that this is not something most tests care about though, so I'd probably be on the side of adding an option to preserve the single-process isolation.

I'm more worried about memory corruption creeping from another test though, but compiling them with address-sanitizer could be a reasonable compromise?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
2 participants
You can’t perform that action at this time.