Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Documented usage of bbench.

Renamed README to indicate Markdown syntax.
Improved documentation on example output.
  • Loading branch information...
commit be16d69e1f7892a57bde13e477a3f13a0a73ab33 1 parent c16bc07
Pistos authored
Showing with 100 additions and 34 deletions.
  1. +0 −34 README
  2. +100 −0 README.md
View
34 README
@@ -1,34 +0,0 @@
-## Dependencies
-
-The R Project: http://www.r-project.org/
-rsruby: http://web.kuicr.kyoto-u.ac.jp/~alexg/rsruby/
-
-## Usage
-
-result = Benchmark.compare_realtime {
- do_something_one_way
-}.with {
- do_it_another_way
-}
-Benchmark.report_on result
-
-See also example.rb for a more comprehensive example.
-
-## Example Output
-
-....................
-Set 1 mean: 0.484 s
-Set 1 std dev: 0.098
-Set 2 mean: 0.469 s
-Set 2 std dev: 0.088
-p.value: 0.601661885634415
-W: 220.0
-The difference (-3.2%) IS NOT statistically significant.
-
-## Help, etc.
-
-irc.freenode.net#mathetes or http://mibbit.com/?server=irc.freenode.net&channel=%23mathetes
-
-## Repository
-
-git clone git://github.com/Pistos/better-benchmark.git
View
100 README.md
@@ -0,0 +1,100 @@
+# Better Benchmark
+
+Statistically correct benchmarking for Ruby.
+
+## Dependencies
+
+The R Project: http://www.r-project.org/
+rsruby: http://web.kuicr.kyoto-u.ac.jp/~alexg/rsruby/
+
+## Usage
+
+### Comparing code blocks
+
+result = Benchmark.compare_realtime {
+ do_something_one_way
+}.with {
+ do_it_another_way
+}
+Benchmark.report_on result
+
+See also example.rb for a more comprehensive example.
+
+### Comparing git revisions
+
+#### With a test script (recommended)
+
+To test two revisions of a library, create a simple runner script:
+
+ # runner.rb
+ require 'mylib'
+
+ class TestQuick
+ def initialize
+ # initialization...
+ end
+
+ def run
+ Benchmark.write_realtime( '/home/pistos/tmp' ) do
+ 5000.times do
+ # do something with your lib
+ end
+ end
+ end
+ end
+
+ t = TestQuick.new
+ t.run
+
+Then run the bbench script, passing two git revisions:
+
+ bbench -r 6e84dd5 -r ed1e7c6 -d ~/tmp -- -Ilib test-quick.rb
+
+#### Without altering or writing new code
+
+You can also test two revisions by running some already-existing script,
+such as a file in your test suite:
+
+ bbench -r 6e84dd5 -r ed1e7c6 -- -Itest -Ilib test/test_something.rb
+
+Be aware, however, that this may produce unnecessarily variant timings due to
+wide variance in the startup time of the Ruby interpreter and script.
+
+## Interpretation
+
+Considering two "things under test", U1 and U2:
+
+### Example 1
+
+ Set 1 mean: 0.216 s
+ Set 1 std dev: 0.023
+ Set 2 mean: 0.187 s
+ Set 2 std dev: 0.020
+ p.value: 0.00287947346770876
+ W: 88.0
+ The difference (-13.5%) IS statistically significant.
+
+This means that the results permit us to conclude that the U2 performs 13.5%
+faster than U1.
+
+### Example 2
+
+ Set 1 mean: 10.968 s
+ Set 1 std dev: 4.294
+ Set 2 mean: 9.036 s
+ Set 2 std dev: 3.581
+ p.value: 0.217562623135379
+ W: 67.0
+ The difference (-17.6%) IS NOT statistically significant.
+
+This means that the results do not permit us to conclude anything; neither that
+U1 performs faster than U2, nor that U2 performs faster than U1, nor even that
+their performance is equivalent.
+
+## Help, etc.
+
+irc.freenode.net#mathetes or http://webchat.freenode.net?channels=mathetes .
+
+## Repository
+
+git clone git://github.com/Pistos/better-benchmark.git
Please sign in to comment.
Something went wrong with that request. Please try again.