Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add parallel benchmarking capability. #15

Merged
merged 8 commits into from Jul 9, 2016

Conversation

Projects
None yet
3 participants
@ldr
Copy link
Contributor

commented Jul 3, 2016

Hi!

I really like benchee but wanted to do some parallel benchmarking of certain code. So I added this in a fork, below shows how it works - maybe you'd like it as well.

It does change output of Benchee.measure from:
[{"bench", [1,2,3,4]}]

into:
[{"bench",[[1,2,3,4],[1,2,3,4]]}]

Where the list is a list of results from separate processes each containing execution times of the passed function. I needed to do this to calc the ips. So had to modify Statistics.statistics for this as well...

regards,

Leon

When passing a flag parallel with value >1 then multiple processes
will be handled for benchmarking provided function.

For example:

iex> Benchee.run %{parallel: 5, time: 10},
                 [{"five", fn -> :timer.sleep(1000) end}]

Benchmarking name: five, parallel: 5, time: 10.0, warmup: 2.0...

Name           ips        average    deviation         median
five            4.98   1003737.24μs     (±0.10%)   1003672.00μs
Add parallel benchmarking capability.
When passing a flag parallel with value >1 then multiple processes
will be handled for benchmarking provided function.

For example:

iex> Benchee.run %{parallel: 5, time: 10},
                 [{"five", fn -> :timer.sleep(1000) end}]

Benchmarking name: five, parallel: 5, time: 10.0, warmup: 2.0...

Name           ips        average    deviation         median
five            4.98   1003737.24μs     (±0.10%)   1003672.00μs
@ldr

This comment has been minimized.

Copy link
Contributor Author

commented Jul 3, 2016

I see the tests fail now, will get back

run_warmup function, warmup
measure_runtimes function, time
end)
end)

This comment has been minimized.

Copy link
@PragTob

PragTob Jul 3, 2016

Member

for readabilities sake I'd probably take this code and put it into a separate function :)

@PragTob

This comment has been minimized.

Copy link
Member

commented Jul 3, 2016

Hi there!

Thanks for this addition, I'd been thinking about adding parallel benchmarking considering the parallel world we live in. I've been on the fence leaning on the "not doing it" side as with parallel processes it's more likely to overload the system, leading to wait time for execution leading to more outliers etc.

Can you tell me what your use case for this was/is? Would be really interested :)

Thanks for being so thorough adding samples and documentation! A test would be nice :)

Tobi

PS: The email you did the commit with doesn't seem to be linked to your github account

@ldr

This comment has been minimized.

Copy link
Contributor Author

commented Jul 3, 2016

Hi,

I needed to benchmark integration tests for a telephony system we wrote - with this system the tests actually interfere with each other (they're using an Ecto repo) and I wanted to see how far I could push the system as a whole. Making this small change to Benchee worked perfectly for what I needed :)

I was first passing a parallel map function that executes the integration test as argument to Benchee, but figured that would not be efficient:

  • unnecessarily starting many processes during the benchmark
  • each iteration within Benchee, all parallel processes start at exactly the same time which causes resource starvation where this new way spreads it out better over time

I've added my other email address to my Github account.

I'll put the parallel map in a separate function and can add some tests, but do you think altering of the output of Benchmark.measure is acceptable since other people may depend on it ? Was thinking I could return the original format when only one parallel thread was used, but that may also be a bit clumsy.

Thanks!

Leon

@PragTob

This comment has been minimized.

Copy link
Member

commented Jul 3, 2016

@ldr there are no plugins for Benchee that I know of plus we are in 0.x so it should be fine :) However, the way I see it calling List.flatten/1 on the output should transform it to its former output and should still work well with statistics et. al. or what am I missing? I'll also change the internal structure in 0.3.0 from an array with tuples to a hash :)

Also, great to hear of your use case and I'm utterly happy that you found Benchee so easy to adjust - mission accomplished.

@ldr

This comment has been minimized.

Copy link
Contributor Author

commented Jul 3, 2016

Hi yes output through lists.flatten gives same output, except now contains more results so ips calculation in job_statistics had to take amount of parallel processes into account.

I did above changes to fix tests etc.

defp pmap(collection, func) do
# only run func through Task.async when >1 parallel is requiested
case collection do
[arg] ->

This comment has been minimized.

Copy link
@TBK145

TBK145 Jul 6, 2016

Ranges are structs, so this will never match. 1..1 would match.

@PragTob

This comment has been minimized.

Copy link
Member

commented Jul 7, 2016

Hey there, thanks! Dunno if I'll get to it/finish it today but I'll make it happen this weekend if nothing else occurs :)

Thanks to both of you!

Tobi

@PragTob

This comment has been minimized.

Copy link
Member

commented Jul 7, 2016

FYI I merged with master and made minor adjustments on this branch: https://github.com/PragTob/benchee/tree/ldr-feature/add_parallel_bench

I'll give it a couple of test runs etc. when I got time and will then make it land on master :) Thanks!

@ldr

This comment has been minimized.

Copy link
Contributor Author

commented Jul 7, 2016

thank you ! :)

@PragTob

This comment has been minimized.

Copy link
Member

commented Jul 9, 2016

So I'm on this, still a few things to fix up etc. but for comparisons sake I ran the branch with just one parallel against master and with multiple parallel processes to see the difference. 4 core system, under normal working load (browser, music, editor, gui etc. open). Good news is that for the single threaded use case it behaves the same, 2 processes is just a bit slower (expected cause single core boost) and afterwards it degrades as somewhat expected. Also the relative performance between benchmarks stays somewhat the same, or well it even gets more apparent lots of times.

tobi@happy ~/github/benchee $ mix run samples/run.exs # vintage
Benchmarking flat_map...
Benchmarking map.flatten...

Name                  ips        average    deviation         median
map.flatten       1274.04       784.90μs    (±14.05%)       756.00μs
flat_map           881.52      1134.40μs     (±6.57%)      1164.50μs

Comparison: 
map.flatten       1274.04
flat_map           881.52 - 1.45x slower
tobi@happy ~/github/benchee $ git checkout -
Switched to branch 'ldr-feature/add_parallel_bench'
tobi@happy ~/github/benchee $ mix run samples/run.exs # parallel 1
Benchmarking flat_map...
Benchmarking map.flatten...

Name                  ips        average    deviation         median
map.flatten       1276.37       783.47μs    (±12.28%)       759.00μs
flat_map           878.60      1138.17μs     (±6.82%)      1185.00μs

Comparison: 
map.flatten       1276.37
flat_map           878.60 - 1.45x slower
tobi@happy ~/github/benchee $ mix run samples/run_parallel.exs # parallel 2
Benchmarking flat_map...
Benchmarking map.flatten...

Name                  ips        average    deviation         median
map.flatten       1230.53       812.66μs    (±19.86%)       761.00μs
flat_map           713.82      1400.92μs     (±5.63%)      1416.00μs

Comparison: 
map.flatten       1230.53
flat_map           713.82 - 1.72x slower
tobi@happy ~/github/benchee $ mix run samples/run_parallel.exs # parallel 3
Benchmarking flat_map...
Benchmarking map.flatten...

Name                  ips        average    deviation         median
map.flatten       1012.77       987.39μs    (±29.53%)       913.00μs
flat_map           513.44      1947.63μs     (±6.91%)      1943.50μs

Comparison: 
map.flatten       1012.77
flat_map           513.44 - 1.97x slower
tobi@happy ~/github/benchee $ mix run samples/run_parallel.exs # parallel 4
Benchmarking flat_map...
Benchmarking map.flatten...

Name                  ips        average    deviation         median
map.flatten        954.88      1047.25μs    (±34.02%)       957.00μs
flat_map           452.38      2210.55μs    (±21.05%)      1914.00μs

Comparison: 
map.flatten        954.88
flat_map           452.38 - 2.11x slower
tobi@happy ~/github/benchee $ mix run samples/run_parallel.exs # parallel 12
Benchmarking flat_map...
Benchmarking map.flatten...

Name                  ips        average    deviation         median
map.flatten        296.63      3371.18μs    (±57.60%)      2827.00μs
flat_map           186.96      5348.74μs    (±42.14%)      5769.50μs

Comparison: 
map.flatten        296.63
flat_map           186.96 - 1.59x slower

Of course, overloading the system with 12 processes is very contra productive :D

@PragTob PragTob merged commit b72a988 into bencheeorg:master Jul 9, 2016

@PragTob

This comment has been minimized.

Copy link
Member

commented Jul 9, 2016

🎉

Thank you once again for contributing back your changes, enhancing Benchee and enabling other people to use it for some stress test benchmarking or whatever they want :) Dunno if I'll manage but I plan/hop to release Benchee this weekend or next week.

@ldr

This comment has been minimized.

Copy link
Contributor Author

commented Jul 11, 2016

Cool, I see it's in master :) good luck with release, i'll change my local deps to your master again :)

@ldr ldr deleted the ldr:feature/add_parallel_bench branch Jul 11, 2016

@PragTob

This comment has been minimized.

Copy link
Member

commented Jul 11, 2016

@ldr if I et myself together, release will be today - at the latest should be end of week :)

@PragTob

This comment has been minimized.

Copy link
Member

commented Jul 11, 2016

@ldr 0.3.0 is live if you want to switch back to using hex.pm :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.