Switch branches/tags
Nothing to show
Find file History
Pull request Compare This branch is 83 commits behind acangiano:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


The Ruby Benchmark Suite is a set of Ruby benchmarks for testing the various available Ruby implementations (MRI, Yarv, JRuby, Rubinius, IronRuby, MacRuby, Ruby Enterprise Edition, MagLev, etcetera). The aim of this project is for it to become a useful tool for VM implementers and a standard suite of benchmarks for periodical shootouts amongst Ruby implementations.

To obtain the suite, simply run:

    git clone git://github.com/acangiano/ruby-benchmark-suite.git
To run the whole suite of benchmarks with Ruby MRI (i.e. 'ruby' in your path), you can run:

    rake run_all

Use RUBY_VM to specify a different Ruby VM. For example:

    rake RUBY_VM="jruby1.1.6 -J-server"
This way you can specify the exact command, including the optimization parameters you may wish to include.

If you'd like to run a single benchmark, you can do so by executing rake run_one and specifying the file location with FILE. For example:

    rake run_one FILE="micro-benchmarks/bm_mergesort.rb"
By default each benchamrk is run 5 times in order to obtain "stastically meaningful" results. You can overwrite this setting by assigning a value to ITERATIONS. Likewise, you can overwrite the default timeout (300 seconds) with TIMEOUT. This is the maximum amount of time allocated for a set of iterations. For example, by default TIMEOUT is 300 and ITERATIONS is 5, hence each iteration can take up to a maxium of 60 seconds to complete. If multiple input sizes are being tested in a benchmark, they'll each be allocated this amount of time in seconds (before timing out). Individual benchmarks may overwrite the global values specified in Rakefile.rb or passed to the rake command as follows:


The report generated in the main folder will be a .csv file whose name has been automatically determined based on the timestamp and Ruby VM in uae. If you'd like to overwrite this, assign a value to REPORT:

   rake REPORT="myreport.csv"
Of course, you are free to combine several parameters amongst FILE, RUBY_VM, ITERATIONS, TIMEOUT and REPORT, as needed.
Please note however that FILE will only work with the run_one task.

The suite is cross-platform, but avoid paths with spaces if you are running it under Windows.

Run rake --tasks to see more options.

The Ruby Benchmark Suite relies on the fact that any exception raised at runtime by a given VM will be rescued and the error message reported within the .csv report. If a less mature RUBY VM encounters a segmentation fault during the execution of a benchmark, the execution of that file terminates before the error can be rescued, and will therefore not appear within the report. Similarly, if due to a bug in a VM, rescuing a given exception is not succesfull, the benchmark at hand will not appear in the report.

What this means in pratice is that certain Ruby VMs may generate a few less entries in the report, when compared to the report produced by Ruby MRI. Comparing the two lists is an easy way to see which tests aborted before they were able to make it into the report. In the case of exceptions which weren't rescued due to a bug in the VM, you'll see the exceptions in the shell where you run rake from, so that you can then add them manually to the report. When running the suite through rake, segmentation faults in individual benchmarks may be silent. To investigate missing entries from the report that didn't show an error message in the shell, you can bypass rake altogether. For example, if the Ruby VM 'myrubyvm' fails to include bm_mergesort.rb in the report, then you can verify what the issue is by running something along the lines of: myrubyvm -myoptions micro-benchmarks/bm_mergesort.rb 300 5 myreport.txt, where the last three parameters must be the timeout, iterations and report filename respectively.

Currently the suite is setup to use the hitimes gem, if it is installed, for more accurate timing on windows/linux.
Also if you want the RSS sizes to show up on windows you'll need to first install the sys-proctable gem.

The mailing list for the project is located at: http://groups.google.com/group/ruby-benchmark-suite