Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Specs fail with RSpec >= 3.1 #18
It's because the reported error message is
The diff between RSpec 2.14 and 3.4 is well over 2000 commits, so it's hard to guess what the cause might be if all we know is "something since 2.14 caused this".
Maybe someone who works on rspec-given could use
Once we have that, it'll be much easier to address. I can help figure out if it's a bug in RSpec or invalid assumption being made here in rspec-given, or something else.
For instance, the very first bisect fails between the two release tags, because running
So here's what I'd suggest you do
You may have to repeat step 4 several times if you're bisecting over many versions. It's worth noting rspec-dev has a rake task to checkout all repositories to a specific version (
I can see two ways to do this...
Since you are aren't trying to bisect across multiple RSpec minor versions, and instead are just trying to bisect all changes that occurred after 3.3.0 was released and before 3.4.0, you should be able to run the bisect from rspec/rspec-core@aaf2964 (the commit that bumped rspec-core to 3.4.0.pre -- just after 3.3.0 was released, and before any "real" changes were made) to rspec/rspec-core@0ab632a (the commit before 3.4.0.pre was bumped to 3.4.0, and after any "real" changes were made). In the other rspec libs, you'll just have to pin them to a commit where the version is 3.4.0.pre, like so:
gem 'rspec-core', path: "/path/to/your/git/clone" gem 'rspec-expectations', github: "rspec/rspec-expectations", ref: "00315bebcf79d9bc40b22928c39e5791c8aaa923" gem 'rspec-mocks', github: "rspec/rspec-mocks", ref: "ad03a31ab61a845550684e5de0274b2568dc2a9c" gem 'rspec-support', github: "rspec/rspec-support", ref: "30b4a01dd039063ce9593e2094062c6f8a0eda34"
The commits I have listed above are the last commit in each repo where the version was 3.4.0.pre.
The other way would be to remove rubygems/bundler from loading RSpec. The version numbers aren't used at runtime by RSpec at all, and if you just add the
Let me know if you need more help, @searls.
Okay, below will look very confusing because the failing test is a test of RSpec's own output using RSpec, so each failure includes a full expected RSpec failure, with the failing assertion being to look for a regex match (as @jkowens fixed #19 )
Prior to rspec/rspec-core@aadd339
Next, I intentionally broke the two tests on the last good commit (parent to rspec/rspec-core@aadd339) to see what the failure message looks like when the rspec-given test is happy:
So fundamentally, it looks like rspec/rspec-core@aadd339 changed the expected behavior of how RSpec prints rspec-given failures from this:
Obviously, Jim designed rspec-given to print the former, and it's much more useful to the end-user than the latter, so I do consider this to be broken from our perspective (though not necessarily a thing that should be fixed in rspec; I don't know yet). However, it does mean that I don't think it'd be prudent to simply shut up the failure by merging #19, either, so I'll be closing it (sorry @jkowens!).
Can you provide any additional wisdom @myronmarston while I dig into the change from the rspec-core side?
Thanks for digging in @searls. I just realized what's going on. The detail you provided above definitely helped.
Before the change in rspec-core, RSpec had a very simple rule for what line of code to look for: it looked for the first backtrace line that matched the file path of the spec file containing the failing example. We don't want to print the code from the first line of the backtrace because that would usually be a line in rspec-expectations where the expectation failure raised an error. This usually worked but not always. For example, if you defined a global
In that change, RSpec became much smarter:
This is affecting rspec-given because rspec-given is a test framework, with test-framework code in the
It should be noted that I do not expect this issue to affect rspec-given end-users at all, because when an rspec-given user uses rspec-given, they do not have rspec-given's source code in their
The solution is to change your RSpec configuration in your spec_helper.rb file so that it removes
RSpec.configure do |config| config.project_source_dirs -= ["lib"] end
Sorry for the confusion, and not realizing the problem sooner!
@searls I believe
Once I did this, the number of failures was reduced from 2 to 1.
The expected output is almost achieved, but there is a slight difference: