Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lockfree benchmarks #194

Merged
merged 9 commits into from
Dec 2, 2020
Merged

Conversation

Sudha247
Copy link
Contributor

This patch contains benchmarks for existing structures in the lockfree library. All lockfree benchmarks are labelled with a lockfree_bench tag.

Benchmark for a data structure contains a sequential and parallel version. Parallel version splits the available work equally to all domains.

@Sudha247
Copy link
Contributor Author

Sudha247 commented Nov 20, 2020

Some results for these benchmarks:
Time
image

Speedup
image

@kayceesrk
Copy link
Contributor

This looks useful. Are the non-scaling benchmarks expected not to scale?

@Sudha247
Copy link
Contributor Author

While I wouldn't expect linear scaling in the non-scaling benchmarks, the current numbers need investigation.

@ctk21
Copy link
Member

ctk21 commented Nov 30, 2020

One thing with these Lockfree benchmarks is that the 'time' and 'speedup' don't tie out with how many of the papers in the area are reporting their numbers. Is it possible to also report the numbers in terms of throughput (e.g. number of transactions per second) or in terms of time per transaction?

@kayceesrk
Copy link
Contributor

kayceesrk commented Dec 2, 2020

I assume that these benchmarks don't have much parallelisable work (except the hash table). For example, consider the benchmark for lock-free queues with a producer and consumer pushing items through the queue. It is unclear whether to expect speedup by splitting the items among multiple producers and consumers, but using the same shared queue. There is increased contention on the shared queue, and there isn't any useful parallelisable work.

Given this, adding some commentary in the top of the test program files would help someone looking at these numbers in the future. I am happy to merge this PR as is, but let's open an issue to track the task to add commentary to these benchmarks. Is this acceptable @Sudha247?

@Sudha247
Copy link
Contributor Author

Sudha247 commented Dec 2, 2020

Yeah, that sounds good! Like Tom mentioned above, it might be useful to measure throughput vs number of cores rather than speedup. ocaml-multicore/saturn#6 has some details about these benchmarks, perhaps we could open a placeholder issue on sandmark too.

@kayceesrk
Copy link
Contributor

@Sudha247 sounds good. Can you open the placeholder issues? I'll merge this PR now.

@kayceesrk kayceesrk merged commit a05940c into ocaml-bench:master Dec 2, 2020
@Sudha247 Sudha247 mentioned this pull request Dec 3, 2020
stedolan pushed a commit to stedolan/sandmark that referenced this pull request Jun 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants