Test Benchmark

A ruby gem and rails plugin to show you how slow your Test::Unit tests run. Useful for troubleshooting/isolating slow tests. Available from http://github.com/timocratic/test_benchmark/

Sample output:

  7.124 test_destroy(FeedTest)
  7.219 test_create(FeedTest)
  7.646 test_subscribe_to_auto_discovery(FeedTest)
  9.339 test_auto_discover_updates_url(FeedTest)
  9.543 test_find_or_create_by_auto_discover_url(FeedTest)
  15.780 test_import_from_opml(FeedTest)

Install and enable

As a Rails Plugin

./script/plugin install git://github.com/timocratic/test_benchmark.git

As a Ruby gem

Install

• Update to RubyGems 1.2.0+ before proceeding!!
• gem sources -a http://gems.github.com (you only have to do this once)
• sudo gem install timocratic-test_benchmark

Enable

• require ‘rubygems’
• require ’test_benchmark

As a (frozen) gem in rails

I recommend this method. It makes updating simple, and if I had more config in the future, that will be easily done only in your test environment

in config/environments/test.rb

config.gem "timocratic-test_benchmark", :lib => 'test_benchmark', :source => 'http://gems.github.com', :version => '0.4.2' #update to whatever version number is current

from the console

RAILS_ENV=test GEM=timocratic-test_benchmark rake gems:install
RAILS_ENV=test GEM=timocratic-test_benchmark rake gems:unpack #optional, but suggested step - 'freezing' the gem

Options and disabling temporarily

By default the top 15 slowest tests are output to the console (and in Rails the whole list is dumped to test.log). To see the full dump in your console (as well as get a per-file/suite breakdown) set the env variable BENCHMARK to ‘full’:

BENCHMARK=full rake test

To disable completely, run with it set to ‘false’ instead.

You can also customize the number of results through the Test::Unit::UI::Console::TestRunner.set_test_benchmark_limits(set_display_limit, set_suite_display_limit) method. For example, putting this line in your test_helper.rb file will output a summary of 5 slowest tests and 3 slowest test classes:

Test::Unit::UI::Console::TestRunner.set_test_benchmark_limits(5, 3)

And this will disable output entirely:

Test::Unit::UI::Console::TestRunner.set_test_benchmark_limits(0, 0)

#Newly Added Feature:

Option to fail the build based on the set threshold limit on benchmarks.

This option is newly added to this gem in order to fail the build if one of the top 15 slowest test’s benchmarks shows a result of exceeding the maximum time(threshold) that a test can take to run.

In order to fail the build based on threshold, First, we need to set the maximum time (in seconds) that any test can take to complete by putting this line in your test_helper.rb file.

Sample:

Test::Unit::UI::Console::TestRunner.capture_benchmarks_from_display_exceeding(2) # 2 -> max_time_in_seconds

Then in the rake file, add a task something like the following and invoke this task as the last target (after invoking all the test targets) as part of the regular build.

Sample:

require 'fileutils'
namespace :benchmark do
  desc "Make sure that the benchmarks are within set threshold limits"
  task :validate do
    unless File.zero?("./log/test_benchmark.txt")
      puts "Error: Failed Benchmark Test. Details are as follows:" 
      benchmark_file = File.open("./log/test_benchmark.txt", 'r') 
      benchmark_file.each { |line| puts line }
      exit 1
    end
    puts "Benchmarking of tests done successfully!"
  end
end