Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove Bench or A generic JHM benchmark interface #2884

Merged
merged 2 commits into from
Jul 25, 2017

Conversation

liufengyun
Copy link
Contributor

@liufengyun liufengyun commented Jul 18, 2017

The old bench/ project is no longer necessary, as it's superseded by:

Web UI: https://liufengyun.github.io/bench/

We can either remove the project bench completely, or change it in to a generic test interface, as this PR does. This PR will enable developers to run jmh benchmarks in command line:

sbt dotty-bench/jmh:compile
sbt dotty-bench/pack
bench/target/pack/bin/bench  hello.scala    # arguments are the same as compiler options

Note: The separation of benchmarking into a separate repo is justified by the fact that (1) we need to test history of Dotty, thus the test driver has to be independent of versions of Dotty; (2) Dotty doesn't depend on it; (3) faster development process.

The generated script `bench` accept all compiler options

1. `sbt bench/pack` to generate compiler artifacts & scripts
2. `bench/target/pack/bin/bench examples/hello.scala`
@DarkDimius
Copy link
Contributor

Suggestions:

  • measure in milliseconds;

  • graph multiple lines:

    • average
    • median [50% percentile]
    • 5% percentile
    • 95% percentile
    • 99% percentile
    • trendline for all listed above
  • store raw data for runs, so that we don't need to rerun benchmarks.

  • add a test that is supposed to always be flat, will be used as indicator of stability of machine.

@DarkDimius
Copy link
Contributor

and please, make the Y axis start from 0.

@DarkDimius
Copy link
Contributor

One more suggestion: could you align the x axis across multiple benchmarks? so that one vertical always relates to the same PR on all benchmarks.

@liufengyun liufengyun mentioned this pull request Jul 18, 2017
6 tasks
@DarkDimius
Copy link
Contributor

LGTM

@DarkDimius DarkDimius merged commit b7cafd8 into scala:master Jul 25, 2017
@allanrenucci allanrenucci deleted the benchmarks branch December 14, 2017 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants