Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

RSpec tests break shoulda-matcher inclusion depending on underlying order #627

fj opened this Issue Jun 4, 2012 · 11 comments


None yet
3 participants

fj commented Jun 4, 2012

We're observing the following issue:

  1) AccessToken
     Failure/Error: it { should validate_presence_of :application }
       undefined method `validate_presence_of' for #<RSpec::Core::ExampleGroup::Nested_82:0x00000010bc8b20>
     # ./spec/models/access_token_spec.rb:5:in `block (2 levels) in <top (required)>'

This error repeats each time there is any use of a shoulda-matcher. For us, the error started appearing when we upgraded to RSpec 2.10.1, and wasn't present in RSpec 2.8. We went back to version 2.9 and it also had the same issue.

It seems to be related to filesystem ordering, because it appears on OSX and not Ubuntu. When Ubuntu users run the files in the OSX order (by pasting the printed list generated from bundle exec rake and running it directly), they experience the errors too. Likewise, if an OSX user pastes the Ubuntu order directly, it will work again for them.

Any hints to narrow down where the problem might be occurring? I'm seeing other people report this too in a number of places, including on shoulda itself.

We do require relevant files from tests, but as far as I know that is the only thing that could be mutating the runtime state in such a way as to affect this, and a require shouldn't affect whether the matchers are present or not.


justinko commented Jun 6, 2012

The shoulda issue has "RSpec-Rails 2.8.1" in the title. Can you verify the error occurs in 2.8.1 and not 2.8.0 to help narrow down the regression? Thanks.


fj commented Jun 6, 2012

Pasting the OSX order into Ubuntu causes errors, but they seem to be a superset of the RSpec 2.10.1 problems (there were 91 failures in 2.10.1, and 108 failures in 2.8.1). However, now running bundle exec rake also causes failures instead of passing on Ubuntu users doing 2.8.1, but it's a different set of failures.

So to summarize:

environmentext4fs orderhfs+ order
on ext4fson hfs+on ext4fson hfs+
rspec-rails 2.8.0passingpassingpassingpassing
rspec-rails 2.8.1passing[not tested]108 errors[not tested]
rspec-rails 2.9.0passing17 errors91 errors91 errors
rspec-rails 2.10.1passingpassing91 errors91 errors

Tests were executed as bundle exec rake to get the native order, although we tried a number of other things (e.g. bundle exec rspec spec). In all cases, when we tried executing individual test files to reproduce missing-validation errors, they pass.


fj commented Jun 6, 2012

Also, I should note that I have tried a number of times to reduce this to a minimal failing project without success. That is, I wanted to make a simple project that had these properties:

  • Uses Rails 3.2.
  • Uses rspec-rails 2.10.1.
  • Uses shoulda-matchers.
  • Uses a matcher from shoulda-matchers.
  • Has two test files.
  • Fails to work for one ordering of the files (e.g. [a_spec.rb, b_spec.rb]) but passes for the other (e.g. [b_spec.rb, a_spec.rb]).

but I wasn't successful in making a project that did that.

I am wondering if that isn't a strong sign that this is an internal issue of some kind, specific to our project, and that RSpec may just be coincidentally involved.


justinko commented Jun 6, 2012

You get no errors when running an individual spec file. And from what I understand, you're not loading the Rails env (the way to go, IMO 😄).

So this error occurs when you load the Rails env. Can you paste what you have in an individual spec file (and also your "fast spec helper"), and also your "normal" spec_helper.rb?


fj commented Jun 7, 2012

Yes, there are two spec-helpers -- one that loads Rails and one that doesn't, as you said:

# spec/spec_helper_with_rails.rb

require 'spec_helper'

ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../../config/environment", __FILE__)
require 'rspec/rails'
Dir["#{File.dirname __FILE__}/support/rails/**/*.rb"].each { |f| require f }

RSpec.configure do |config|
  config.use_transactional_fixtures = true
# spec/spec_helper.rb

require 'simplecov'

# Add project root as the default search path.
$:.unshift "#{File.dirname __FILE__}/.."

RSpec.configure do |config|
  config.mock_with :rspec

# Include support files.
Dir["#{File.dirname __FILE__}/support/base/**/*.rb"].each { |f| require f }

fj commented Jun 7, 2012

Can you paste what you have in an individual spec file ...

Do you want the whole file? If so, what kind of spec file are you looking for? (There are 100+ spec files and I assume you don't want to look through all of 'em. :) )


justinko commented Jun 7, 2012

Do you have shoulda-matchers in the :development bundle group? If so, try putting it only in the :test group.


fj commented Jun 7, 2012

I don't, no; it's in :test only. Here's the whole Gemfile:

source 'http://rubygems.org'

gem 'rails', '3.2.5'

# analytics and monitoring  
gem 'airbrake'
gem 'newrelic_rpm'

# DB gems
gem 'pg'

# miscellaneous
gem 'rack-ssl', :require => 'rack/ssl'

# presentation
gem 'jquery-rails', "2.0.2" 
gem 'haml', "3.1.4"
gem "kaminari", "0.13.0"

# asset pipeline
group :assets do
  gem "yui-compressor"
  gem "sass-rails"
  gem "compass-rails"
  gem "compass-susy-plugin", :require => 'susy'

# emails
gem 'aws-sdk'

# asynchronous processing   
gem 'resque'

# state machine
gem "state_machine", "1.1.2"

# authentication & authorization
gem 'devise'

# see:
# https://github.com/intridea/omniauth/issues/562
# https://github.com/rack/rack/issues/322
gem 'omniauth'
gem "omniauth-twitter"
gem 'cancan', "1.6.4"

# feature flipping / continuous deployment
gem 'ruby_flipper'

gem 'bcrypt-ruby', :group => [:development, :production, :staging]

# twitter integration
gem 'twitter'
gem 'encryptor'

# handle heroku integration
gem 'heroku_san'

# geocoding
gem 'geocoder'

# i18n
gem 'countries'
gem 'phony'

# clouds
gem "fog", :git => 'git://github.com/fog/fog.git'

# image storage and manipulation
gem "rmagick", "2.13.1"
gem "carrierwave"

group :deployment do
  gem "thin"

group :development do
  gem "foreman"

group :development, :test do
  gem 'haml-rails'
  gem 'rails3-generators'
  gem 'rspec-rails', "~> 2.10"
  gem 'ruby-debug19'

group :test do
  gem 'capybara'
  gem 'cucumber-rails'
  gem 'database_cleaner'
  gem 'factory_girl', "1.3.3"
  gem 'shoulda-matchers'
  gem 'simplecov'
  gem 'timecop'
  gem 'rack-oauth2'
  gem 'webmock'
  gem 'email_spec'

justinko commented Jun 7, 2012

Sorry John, but I've repeatedly tried to reproduce this issue with the info given. I've tried being explicit with the file order. I've tried using your exact Gemfile.

Since you're not able to reproduce this in a "fresh" rails app, I'm going to say the problem occurs in the code of the offending Rails app.

Going to close this. If you find the cause, please report back here.

P.S. If you are comfortable with it, I can help you debug this app. You know how to contact me :)

@justinko justinko closed this Jun 7, 2012


fj commented Jun 7, 2012

Understandable -- I couldn't reproduce it myself either. I'm guessing there's a hidden ordering dependency in one or more of the requires.

In the above Gemfile, the exact problem can be reproduced by moving the test group before the development & test group.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment