Conversation
|
Do we want an "even wider" bench (eg 1000 or 10000 tasks)? 100 is just not that many. |
|
Just noting that |
17fa1b5 to
2a476c2
Compare
I added an instance of the scaling benchmark with 1024 tasks. I'd guess that any issues with scaling threads should be visible at that size.
Yeah, it's a bit unfortunate, but I do suspect mostly it won't be noticed or used. I've added a comment to |
2a476c2 to
7fdd45a
Compare
7fdd45a to
0b07402
Compare
This PR improves the benchmarking suite of Shuttle.
(A) The benchmarks can now be run without vector-clock overhead by using the
bench-no-vector-clocksfeature. This feature is now enabled in the benchmarking Github action.(B) The benchmarks have also been expanded and parameterized by the number of tasks and number of total events across all threads. More specifically:
create.rshas been added to give a rough gauge of thread creation time and Shuttle startup overheadcounter.rsandlock.rshave been parameterized and set to run in a "narrow" [5 tasks] and "wide" [100 tasks] configuration. The number of total events/operations remains constant [10000 events] across these configurations (the number of events per task is computed from the tasks and the total number of events).3*5*2=30) to be run in a reasonable amount of time. As such, statistical comparisons on individual data-points from these scaling benchmarks are not particularly informative. Unfortunately, Criterion doesn't have an automatic way to visualize trends, so the reports generated for these are not easy to interpret. In the future, it might be useful to generate custom reports from the raw data instead.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.