some examples in Readme are failing

[exocode]

Describe the problem

I am learning this gem and tried the examples in the README.
And just wann know if this is either:

• a configuration issue on my side
• a typo in the docs
• a bug

Steps to reproduce the problem

    it "how many objects are left" do
      #  And you can also check how many objects are left when expectation finishes to ensure that GC is able to collect them.
      expect { ["foo", "bar", "baz"].sort[1] }.to perform_allocation(3).and_retain(3)
    end

    it "perform_constant" do
      sizes = bench_range(8, 100_000) # => [8, 64, 512, 4096, 32768, 100000]
      number_arrays = sizes.map { |n| Array.new(n) { rand(n) } }
      expect { |n, i| number_arrays[i].max }.to perform_linear.in_range(8, 100_000).ratio(2)
    end

Actual behaviour

expected block to perform allocation of 3 objects and retain 3 objects, but allocated 6 objects and retained 6

  0) Performance testing examples Allocation how many objects are left
     Failure/Error: expect { ["foo", "bar", "baz"].sort[1] }.to perform_allocation(3).and_retain(3)
       expected block to perform allocation of 3 objects and retain 3 objects, but allocated 6 objects and retained 6
     # ./spec/benchmark/sanity_check_spec.rb:135:in `block (3 levels) in <top (required)>'
     # ./spec/rails_helper.rb:115:in `block (3 levels) in <top (required)>'
     # ./spec/rails_helper.rb:114:in `block (2 levels) in <top (required)>'


NoMethodError: undefined method `max' for nil:NilClass

  0) Performance testing examples Complexity perform_constant
     Failure/Error: expect { |n, i| number_arrays[i].max }.to perform_linear.in_range(8, 100_000).ratio(2)

     NoMethodError:
       undefined method `max' for nil:NilClass
     # ./spec/benchmark/sanity_check_spec.rb:115:in `block (4 levels) in <top (required)>'
     # ./spec/benchmark/sanity_check_spec.rb:115:in `block (3 levels) in <top (required)>'
     # ./spec/rails_helper.rb:115:in `block (3 levels) in <top (required)>'
     # ./spec/rails_helper.rb:114:in `block (2 levels) in <top (required)>'

Expected behaviour

Go green

Describe your environment

• OS version: 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64
• Ruby version: ruby '2.7.2'
• RSpec::Benchmark version: rspec-benchmark (0.6.0)

[piotrmurach]

Hi Jan,

Thanks for trying out rspec-benchmark. 👍

Given the options, I believe this is

a configuration issue on my side

I can see that you're testing inside the Rails app from the error messages. Please note that all the examples in the readme are tested using Ruby without any frameworks. So I'd encourage you to set up the following file outside of the Rails context for your future explorations:

require "bundler/inline"

gemfile do
  source "https://rubygems.org"
  gem "rspec-benchmark"
end

RSpec.describe do
  include RSpec::Benchmark::Matchers
  ...
end

The first test works as expected.

it "how many objects are left" do
  expect { ["foo", "bar", "baz"].sort[1] }.to perform_allocation(3).and_retain(3)
end

So why is your expectation failing? My guess is that Rails autoloads and potentially monkey patches some behaviour resulting in more objects being created. For example, running Object.new.methods.size in Ruby 2.7 irb session returns 58. The same code run in Rails 7 bare-bones app console returns 87.

The second test fails.

it "perform_constant" do
  sizes = bench_range(8, 100_000) # => [8, 64, 512, 4096, 32768, 100000]
  number_arrays = sizes.map { |n| Array.new(n) { rand(n) } }
  expect { |n, i| number_arrays[i].max }.to perform_linear.in_range(8, 100_000).ratio(2)
end

However, this is a configuration issue. The expectation range generates more steps than the number_arrays has elements, hence the error on nil. When you generate number_arrays you need to ensure that it matches the range you run the performance expectation with. Specifically, the bench_range(8, 100_000) generates [8, 64, 512, 4096, 32768, 100000] which uses ratio of 8 by default. However, the in_range(8, 100_00).ratio(2) generates [8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 100000] which uses ratio of 2. So to fix the test, one way is to update bench_range to use ratio: 2 as well and use generated sizes to specify the expectation range.

The following test passes:

it "perform_constant" do
  sizes = bench_range(8, 100_000, ratio: 2)
  number_arrays = sizes.map { |n| Array.new(n) { rand(n) } }
  expect { |n, i| number_arrays[i].max }.to perform_linear.in_range(sizes.first, sizes.last).ratio(2)
end

The benchark-trend has more explanation as to how the API works.