Skip to content
Input-focused benchmarking for Ruby
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
lib/benchmark
test
.gitignore
.travis.yml
CHANGELOG.md
Gemfile
LICENSE.txt
README.md
Rakefile
benchmark-inputs.gemspec

README.md

benchmark-inputs

Input-focused benchmarking for Ruby. Given one or more blocks and an array of inputs to yield to each of them, benchmark-inputs will measure the speed (in invocations per second) of each block. Blocks which execute very quickly, as in microbenchmarks, are automatically invoked repeatedly to provide accurate measurements.

Motivation

I <3 Fast Ruby. By extension, I <3 benchmark-ips. But, for some use cases, benchmark-ips doesn't let me write benchmarks the way I'd like. Consider the following example, using benchmark-ips:

require "benchmark/ips" ### USING benchmark-ips (NOT benchmark-inputs)

STRINGS = ["abc", "aaa", "xyz", ""]
Benchmark.ips do |job|
  job.report("String#tr"){ STRINGS.each{|s| s.tr("a", "A") } }
  job.report("String#gsub"){ STRINGS.each{|s| s.gsub(/a/, "A") } }
  job.compare!
end

The calls to STRINGS.each introduce performance overhead that skews the time measurements. The less time the target function takes, the more relative overhead, and thus more skew. For a microbenchmark this can be a problem. A possible workaround is to invoke the function on each value individually, but that is more verbose and more error-prone:

require "benchmark/ips" ### USING benchmark-ips (NOT benchmark-inputs)

s1, s2, s3, s4 = ["abc", "aaa", "xyz", ""]
Benchmark.ips do |job|
  job.report("String#tr") do
    s1.tr("a", "A")
    s2.tr("a", "A")
    s3.tr("a", "A")
    s4.tr("a", "A")
  end
  job.report("String#gsub") do
    s1.gsub(/a/, "A")
    s2.gsub(/a/, "A")
    s3.gsub(/a/, "A")
    s4.gsub(/a/, "A")
  end
  job.compare!
end

Enter benchmark-inputs. Here is how the same benchmark looks using this gem:

require "benchmark/inputs" ### USING benchmark-inputs

Benchmark.inputs(["abc", "aaa", "xyz", ""]) do |job|
  job.report("String#tr"){|s| s.tr("a", "A") }
  job.report("String#gsub"){|s| s.gsub(/a/, "A") }
  job.compare!
end

Which prints something like the following to $stdout:

String#tr
  1387268.0 i/s (±0.49%)
String#gsub
  264307.7 i/s (±1.95%)

Comparison:
    String#tr:   1387268.0 i/s
  String#gsub:    264307.7 i/s - 5.25x slower

Benchmarking destructive operations

Destructive operations also pose a challenge for microbenchmarks. Each invocation needs to operate on the same data, but duping the data introduces too much overhead and skew.

benchmark-inputs' solution is to estimate the overhead incurred by each dup, and exclude that from the time measurements. Because the benchmark job already controls the input data, everything can be handled behind the scenes. To enable this, use the dup_inputs option:

require "benchmark/inputs"

Benchmark.inputs(["abc", "aaa", "xyz", ""], dup_inputs: true) do |job|
  job.report("String#tr!"){|s| s.tr!("a", "A") }
  job.report("String#gsub!"){|s| s.gsub!(/a/, "A") }
  job.compare!
end

Which prints out something like:

String#tr!
  1793132.0 i/s (±0.46%)
String#gsub!
  281588.6 i/s (±0.49%)

Comparison:
    String#tr!:   1793132.0 i/s
  String#gsub!:    281588.6 i/s - 6.37x slower

That shows a slightly larger performance gap than the previous benchmark. This makes sense because the overhead of allocating new strings -- previously via a non-bang method, but now via dup -- is now excluded from the timings. Thus, the speed of tr! relative to gsub! is further emphasized.

Limitations

Benchmark.inputs generates code based on the array of input values it is given. Each input value becomes a local variable. While there is theoretically no limit to the number of local variables that can be generated, more than a few hundred may slow down the benchmark. But, because input values are used to represent different scenarios rather than control the number of invocations, this limitation shouldn't pose a problem.

Installation

$ gem install benchmark-inputs

Usage

See the example above, or check out the API documentation.

License

MIT License

You can’t perform that action at this time.