Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Add parallel benchmarking capability. #15
I really like benchee but wanted to do some parallel benchmarking of certain code. So I added this in a fork, below shows how it works - maybe you'd like it as well.
It does change output of Benchee.measure from:
Where the list is a list of results from separate processes each containing execution times of the passed function. I needed to do this to calc the ips. So had to modify Statistics.statistics for this as well...
When passing a flag parallel with value >1 then multiple processes
Thanks for this addition, I'd been thinking about adding parallel benchmarking considering the parallel world we live in. I've been on the fence leaning on the "not doing it" side as with parallel processes it's more likely to overload the system, leading to wait time for execution leading to more outliers etc.
Can you tell me what your use case for this was/is? Would be really interested :)
Thanks for being so thorough adding samples and documentation! A test would be nice :)
PS: The email you did the commit with doesn't seem to be linked to your github account
I needed to benchmark integration tests for a telephony system we wrote - with this system the tests actually interfere with each other (they're using an Ecto repo) and I wanted to see how far I could push the system as a whole. Making this small change to Benchee worked perfectly for what I needed :)
I was first passing a parallel map function that executes the integration test as argument to Benchee, but figured that would not be efficient:
I've added my other email address to my Github account.
I'll put the parallel map in a separate function and can add some tests, but do you think altering of the output of Benchmark.measure is acceptable since other people may depend on it ? Was thinking I could return the original format when only one parallel thread was used, but that may also be a bit clumsy.
@ldr there are no plugins for Benchee that I know of plus we are in 0.x so it should be fine :) However, the way I see it calling
Also, great to hear of your use case and I'm utterly happy that you found Benchee so easy to adjust - mission accomplished.
FYI I merged with master and made minor adjustments on this branch: https://github.com/PragTob/benchee/tree/ldr-feature/add_parallel_bench
I'll give it a couple of test runs etc. when I got time and will then make it land on master :) Thanks!
So I'm on this, still a few things to fix up etc. but for comparisons sake I ran the branch with just one parallel against master and with multiple parallel processes to see the difference. 4 core system, under normal working load (browser, music, editor, gui etc. open). Good news is that for the single threaded use case it behaves the same, 2 processes is just a bit slower (expected cause single core boost) and afterwards it degrades as somewhat expected. Also the relative performance between benchmarks stays somewhat the same, or well it even gets more apparent lots of times.
Of course, overloading the system with 12 processes is very contra productive :D