Mason Bench and benchmarking framework design #15680
Comments
This comment has been minimized.
This comment has been minimized.
|
A good thing to discuss would be the output of |
|
@ronawho has been working with performance testing of some user codes (https://github.com/mhmerrill/arkouda), and may have some design input / requirements here. |
Implementation proposal:Following is a proposed implementation of the benchmarking framework. module BenchMarking {
class Bench {
// number of repititions - maximum 1e9
private int N;
// initialisation
var netDuration;
var netAllocs;
var netBytes;
// to reset timer and allocated memory heaps
proc resetTimer() {}
// to start a timer and initialise memory allocation
proc startTimer() {}
// stops timer and gathers net allocated memory and total time
proc stopTimer() {
// startAllocs, startBytes = initial state of totalAllocs and malloc
// netAllocs, netBytes = final state after running function
duration = timer.finish() - timer.start();
netAllocs += currentMallocs - startAllocs;
netByte += totalAllocs - startBytes;
}
// returns total time
proc totalTime() {}
// returns average time
proc avgTime() {}
// Detection of subbenchmarks ?
// if subbenchmark function is NOT present
proc run() {
Bench.resetTimer();
Bench.startTimer();
//benchfunction here
Bench.stopTimer();
var totalTime = Bench.totalTime();
var avgTime = totalTime / Bench.N;
}
// if subbenchmark is present
// run Benchamrk function once
proc runOnce() {}
// run subbenchmark N times
proc runNTimes() {}
proc main() {}
}
// various metrics to be displayed in result
class BenchMarkMetrics {
// metric - ns/op
proc NsPerOp() { return Bench.netDuration.nanoseconds() / Bench.N; }
// metric - mb/s
proc mbPerSec() {
var bytes = float64(Bench.netBytes);
var N = float64(Bench.N);
return (bytes * N / 1e6) / Bench.duration;
}
// metric - allocs/op
proc allocsPerOp() {
return Bench.netAllocs/Bench.N;
}
// metric - allocedBytes/op
proc allocedBytesPerOp() {
return Bench.netBytes/Bench.N;
}
}
}I hope I have covered the basics for an initial version of the benchmarking framework. Please add new functions or modify existing by editing this comment. |
|
@ankingcodes - Your outline will be helpful as we approach implementation. However, I think we should start at an even higher level: What will this interface look like to a user? Let's start with a benchmark in Chapel's repo and imagine how it would be expressed with the stencil.chplThere's a lot of metadata to support here: Correctness metadata:
Performance metadata:
Multilocale performance metadata:
Here is a reference for how the performance testing works for this test system. For each piece of metadata, we need to decide how it will be expressed. The two main ways to express metadata in
|
@ben-albrecht I believe the reason of using Mason.toml is because of lack of support of |
|
I believe the role of private proc benchSignature(test : borrowed Bench) throws {}I'm not sure about correction testing though. Could we reuse functions of UnitTest for that ? |
|
Another feature that comes to mind is comparing with a previous performance test to get a declaration of |
I am not sure what you mean by this.
Agreed.
I agree, but am not yet sure how
Something like that could work.
I think so. We could just re-use assertions from
👍 I think that'd be a great feature. |
We cannot specify
Sounds good !
Is there any specific reason for suggesting |
|
I'll try to clear out #15695 before working on Bench. It will help me get more familiar with the working of test. |
CSV is ubiquitous for storing 2-dimensional data.
Maybe the metric to be logged can be saved via a function (method) call like: proc benchTest(test : borrowed Bench) throws {
var rate = 0.0;
var avgTime = 0.0;
// Run the benchmark, assign rate and avgTime
test.record(rate, 'Rate (MFlops/s)');
test.record(avgTime, 'Avg time (s)');
} |
|
I'm looking at this issue for the first time and for people in my situation it would really help to have a summary of the current direction for user-facing API in the issue description up top. I can see for example |
|
@ankingcodes - Let's try to "port" what some existing performance tests would look like under I was originally going to point you to tests from Arkouda as an example, but it turns out they are quite complex due to launching a Chapel process and Python process which connects to it. That may be out of the scope of
I don't think we need full examples for all of these, but we should provide at least one full example and have snippets of code showing how we will represent the metadata for each one. This should give a clearer picture of what we are proposing and help raise any design questions that have not yet been considered. |
|
@ben-albrecht @Spartee @krishnadey30 @ronawho @mppf - Some more work on the design has been done. We have a repository consisting of the above mentioned Link to repository : https://github.com/ankingcodes/mason-bench-design |
Summary (updated)
Chapel requires a benchmarking tool inheriting features from
start_test, which would be used with amason benchcommand. Following the design ofmason testand the Unit Test module, Mason packages would have abench/directory which would hold all the benchmark programs. The benchmark programs can also be added to the manifest file (Mason.toml) as follows :Performance metadata and multilocale performance metadata will be expressed in a manifest (TOML) file, i.e.,
Mason.tomlfile.tomlfile with the name same as that of the program. i.eFoo.chplwill have a manifest file namedFoo.toml. This feature would be available after a PR on Support test metadata file for UnitTest outside of mason packages #15695In case of having multiple performance tests in a single file, the performance test functions would be identified using the function signature :
The output would be logged in a
.csvfile which would be used by a latter performance run to compare and declare an improvement/regression.A
mason benchcommand when run inside a Mason package directory would look for performance tests in Mason.toml or thebench/directory. If used outside of Mason package, the test should be given as argument, i.e,mason bench perfTest.chpl, otherwise, mason bench would run only those programs which have a.tomlfile with the same name.We plan to support multilocale performance testing using a
-mlflag.We have planned to keep
mason benchsimilar tostart_testso that an experienced Chapel user can easily transition tomason bench. However, we intend to support graph generation in a later version ofmason bench.Additionally, please check comments below for more details.
This issue is a part of GSoC proposal "Improvements to Mason Package Manager", and it aims to extend the UnitTest module by adding a benchmarking framework and to implement a
mason benchcommand that would be used to run benchmarks.Initial discussions that have taken place in Slack with @ben-albrecht @Spartee and @krishnadey30 can be summarised as follows :
start_test --performanceso that transition fromstart_testtomason benchis easy for an user and loss of features would be minimal.mason benchcommand would resemble tocargo benchcommand. Also,mason benchshould be usable both inside and outside mason packages, similar tomason test.mason benchwould consists of essential options available incargo benchsuch as--no-fail-fast,--offline,--verboseetc.mason benchis called inside a mason package, then it should look for files inside abenchdirectory, else if it's called outside a mason package, it should look for files containing benchmarking functions by crawling through the directories.The text was updated successfully, but these errors were encountered: