Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmarks we might add to the test suite #33564

Open
hfinkel opened this issue Aug 17, 2017 · 11 comments
Open

Benchmarks we might add to the test suite #33564

hfinkel opened this issue Aug 17, 2017 · 11 comments

Comments

@hfinkel
Copy link
Collaborator Author

@hfinkel hfinkel commented Aug 17, 2017

http://parsec.cs.princeton.edu/

https://github.com/graph500/graph500/tree/v2-spec (the current reference implementation requires MPI, but the v2 branch still has serial and OpenMP versions (etc.))

https://github.com/benchmark-subsetting/NPB3.0-omp-C

http://www.highproductivity.org/SSCABmks.htm (this web site does not exist any more, but there seems to be a copy of some of the benchmarks https://github.com/gtcasl/hpc-benchmarks/tree/master/SSCA2v2.2)

@Meinersbur
Copy link
Member

@Meinersbur Meinersbur commented Oct 23, 2018

A proposal list has been added (docs/Proposals/TestSuite.rst) in r345074

@RKSimon
Copy link
Collaborator

@RKSimon RKSimon commented Oct 23, 2018

Roman has been using his image processing suite as part of the AMD Piledriver (BDVER2) scheduler model development, he might be able to advise if this is something that would work well as a test-suite addition:

https://github.com/darktable-org/rawspeed

@LebedevRI
Copy link
Member

@LebedevRI LebedevRI commented Oct 23, 2018

Roman has been using his image processing
Not really processing, decoding.

suite as part of the AMD
Piledriver (BDVER2) scheduler model development, he might be able to advise
if this is something that would work well as a test-suite addition:

https://github.com/darktable-org/rawspeed

It totally could, both as benchmark, and a test, i would love that.
I even brought it up in https://reviews.llvm.org/D46714#1095030

One caveat: it requires a test set, which is kinda heavy - 756M right now,
not something you'd want to put into test-suite repo.
https://raw.pixls.us/data-unique/ (+ there is rsync), we are looking into
putting it into a git repo to simplify syncing.

@Meinersbur
Copy link
Member

@Meinersbur Meinersbur commented Oct 24, 2018

I added it to the proposal list (http://llvm.org/docs/Proposals/TestSuite.html) in r345166.

A data set of that size might also be unsuitable for compiler performance testing because a large portion of the runtime will be I/O (unless the host has a sufficient amount of RAM and we can somehow guarantee it being in the cache).

Instead, could we craft smaller RAW images, or use well-compressible images (e.g. all-gray) that are decompressed on-the-fly?

@LebedevRI
Copy link
Member

@LebedevRI LebedevRI commented Oct 24, 2018

I added it to the proposal list
(http://llvm.org/docs/Proposals/TestSuite.html) in r345166.
Thanks!

A data set of that size might also be unsuitable for compiler performance
testing because a large portion of the runtime will be I/O (unless the host
has a sufficient amount of RAM and we can somehow guarantee it being in the
cache).
I'm not sure i follow. What specifically is the concern here?
The disk IO itself, or that it will be counted as the time consumed by benchmark?
Because the latter is absolutely avoidable by only counting the actual decoding time, not reading time.
(The whole file is already read to memory first, and decoded from memory.)

Instead, could we craft smaller RAW images, or use well-compressible images
(e.g. all-gray) that are decompressed on-the-fly?

Sadly, it's not your typical libpng / libjpeg.
There aren't actual specs for these compression schemes available.
And even then, one would need to implement the compressors,
and then ensure that they correctly roundtrip, etc.
So while it's not unfeasible, just not too feasible..

@Meinersbur
Copy link
Member

@Meinersbur Meinersbur commented Oct 24, 2018

I'm not sure i follow. What specifically is the concern here?
The disk IO itself, or that it will be counted as the time consumed by
benchmark?

The latter

Because the latter is absolutely avoidable by only counting the actual
decoding time, not reading time.
(The whole file is already read to memory first, and decoded from memory.)

Most programs in test-suite, which the easiest to do, have their total execution time measured (timeit.c or linux perf). It's the user time (not wall-clock or kernel), I don't how much I/O has still an influence on it.

I now see that rawspeed's benchmarks (https://github.com/darktable-org/rawspeed/tree/develop/bench) are already using Google Benchmark, which is understood by test-suite's microbenchmark.py.

@LebedevRI
Copy link
Member

@LebedevRI LebedevRI commented Oct 24, 2018

I'm not sure i follow. What specifically is the concern here?
The disk IO itself, or that it will be counted as the time consumed by
benchmark?

The latter
OK, great :)

Because the latter is absolutely avoidable by only counting the actual
decoding time, not reading time.
(The whole file is already read to memory first, and decoded from memory.)

Most programs in test-suite, which the easiest to do, have their total
execution time measured (timeit.c or linux perf). It's the user time (not
wall-clock or kernel), I don't how much I/O has still an influence on it.

I now see that rawspeed's benchmarks
(https://github.com/darktable-org/rawspeed/tree/develop/bench)
Eh, those "benchmarks" are bad; so far, i never really looked into writing small proper benchmarks.

are already using Google Benchmark
Absolutely :)
The actual interesting thing is https://github.com/darktable-org/rawspeed/blob/develop/src/utilities/rsbench/main.cpp

which is understood by test-suite's microbenchmark.py.

@davidbolvansky
Copy link
Collaborator

@davidbolvansky davidbolvansky commented Oct 30, 2018

@Meinersbur
Copy link
Member

@Meinersbur Meinersbur commented Nov 21, 2018

@llvmbot llvmbot transferred this issue from llvm/llvm-bugzilla-archive Dec 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants