FooBarWidget / ruby-benchmark-suite forked from acangiano/ruby-benchmark-suite

The Ruby Benchmark Suite is a set of Ruby benchmarks for testing the various available Ruby implementations (MRI, Yarv, 
JRuby, Rubinius, IronRuby, MacRuby, Ruby Enterprise Edition, MagLev, etcetera). The aim of this project is for it to 
become a useful tool for VM implementers and a standard suite of benchmarks for periodical shootouts amongst Ruby 
implementations.

GETTING THE SUITE
=================
To obtain the suite, simply run:

    git clone git://github.com/acangiano/ruby-benchmark-suite.git
  
  
RUNNING THE SUITE
=================
To run the whole suite of benchmarks with Ruby MRI (i.e. 'ruby' in your path), you can run:

    rake
or
    rake run_all

Use RUBY_VM to specify a different Ruby VM. For example:

    rake RUBY_VM="jruby1.1.6 -J-server"
  
This way you can specify the exact command, including the optimization parameters you may wish to include.

If you'd like to run a single benchmark, you can do so by executing rake run_one and specifying the file location with 
FILE. For example:

    rake run_one FILE="micro-benchmarks/bm_mergesort.rb"
  
By default each benchamrk is run 5 times in order to obtain "stastically meaningful" results. You can overwrite this 
setting by assigning a value to ITERATIONS. Likewise, you can overwrite the default timeout (300 seconds) with TIMEOUT. 
This is the maximum amount of time allocated for a set of iterations. For example, by default TIMEOUT is 300 and 
ITERATIONS is 5, hence each iteration can take up to a maxium of 60 seconds to complete. If multiple input sizes are 
being tested in a benchmark, they'll each be allocated this amount of time in seconds (before timing out). Individual 
benchmarks may overwrite the global values specified in Rakefile.rb or passed to the rake command as follows:

   rake ITERATIONS=3 TIMEOUT=600

The report generated in the main folder will be a .csv file whose name has been automatically determined based on the 
timestamp and Ruby VM in uae. If you'd like to overwrite this, assign a value to REPORT:

   rake REPORT="myreport.csv"
   
Of course, you are free to combine several parameters amongst FILE, RUBY_VM, ITERATIONS, TIMEOUT and REPORT, as needed.
Please note however that FILE will only work with the run_one task.

The suite is cross-platform, but avoid paths with spaces if you are running it under Windows.


MANAGING ERRORS
===============
The Ruby Benchmark Suite relies on the fact that any exception raised at runtime by a given VM will be rescued and the 
error message reported within the .csv report. If a less mature RUBY VM encounters a segmentation fault during the 
execution of a benchmark, the execution of that file terminates before the error can be rescued, and will therefore not 
appear within the report. Similarly, if due to a bug in a VM, rescuing a given exception is not succesfull, the 
benchmark at hand will not appear in the report.

What this means in pratice is that certain Ruby VMs may generate a few less entries in the report, when compared to the 
report produced by Ruby MRI. Comparing the two lists is an easy way to see which tests aborted before they were able to 
make it into the report. In the case of exceptions which weren't rescued due to a bug in the VM, you'll see the 
exceptions in the shell where you run rake from, so that you can then add them manually to the report. When running the 
suite through rake, segmentation faults in individual benchmarks may be silent. To investigate missing entries from the 
report that didn't show an error message in the shell, you can bypass rake altogether. For example, if the Ruby VM 
'myrubyvm' fails to include bm_mergesort.rb in the report, then you can verify what the issue is by running something 
along the lines of: myrubyvm -myoptions micro-benchmarks/bm_mergesort.rb 300 5 myreport.txt, where the last three 
parameters must be the timeout, iterations and report filename respectively.

The mailing list for the project is located at: http://groups.google.com/group/ruby-benchmark-suite