New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specs fail with RSpec >= 3.1 #18

Closed
paddor opened this Issue Jan 1, 2016 · 21 comments

Comments

Projects
None yet
5 participants
@paddor

paddor commented Jan 1, 2016

As you can see for example in this insignificant PR's build, some specs are failing when run using RSpec >= 3.1, in this case RSpec 3.4.

It's because the reported error message is Failure/Error: ::RSpec::Expectations.fail_with(*args) instead of something like Failure/Error: Then { ToBool.new }. I don't see why it does that and don't know how to fix it.

@searls

This comment has been minimized.

Show comment
Hide comment
@searls

searls Jan 11, 2016

Thank you for taking the time to look into this @paddor

searls commented Jan 11, 2016

Thank you for taking the time to look into this @paddor

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Jan 11, 2016

Is this definitely a new issue with 3.4, and not with 3.3 or earlier?

My best guess is that rspec/rspec-core#2088 is somehow the cause of this, although I'm not sure how it could cause it (partially because I don't know how rspec-given works).

Does that help?

myronmarston commented Jan 11, 2016

Is this definitely a new issue with 3.4, and not with 3.3 or earlier?

My best guess is that rspec/rspec-core#2088 is somehow the cause of this, although I'm not sure how it could cause it (partially because I don't know how rspec-given works).

Does that help?

@paddor

This comment has been minimized.

Show comment
Hide comment
@paddor

paddor Jan 11, 2016

@myronmarston I don't know if it's maybe 3.3 or earlier. I just noticed that the Travis CI builds which use RSpec 2.14 succeed.

paddor commented Jan 11, 2016

@myronmarston I don't know if it's maybe 3.3 or earlier. I just noticed that the Travis CI builds which use RSpec 2.14 succeed.

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Jan 11, 2016

The diff between RSpec 2.14 and 3.4 is well over 2000 commits, so it's hard to guess what the cause might be if all we know is "something since 2.14 caused this".

Maybe someone who works on rspec-given could use git bisect against rspec-core's master branch in order to pinpoint the source commit?

Once we have that, it'll be much easier to address. I can help figure out if it's a bug in RSpec or invalid assumption being made here in rspec-given, or something else.

myronmarston commented Jan 11, 2016

The diff between RSpec 2.14 and 3.4 is well over 2000 commits, so it's hard to guess what the cause might be if all we know is "something since 2.14 caused this".

Maybe someone who works on rspec-given could use git bisect against rspec-core's master branch in order to pinpoint the source commit?

Once we have that, it'll be much easier to address. I can help figure out if it's a bug in RSpec or invalid assumption being made here in rspec-given, or something else.

@searls

This comment has been minimized.

Show comment
Hide comment
@searls

searls commented Jan 11, 2016

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Jan 11, 2016

Good to know. The PR I linked to above it the likely culprit, although I'm not able to guess why it caused this problem. git bisect would still be useful as a way to confirm my hunch (or prove it wrong).

myronmarston commented Jan 11, 2016

Good to know. The PR I linked to above it the likely culprit, although I'm not able to guess why it caused this problem. git bisect would still be useful as a way to confirm my hunch (or prove it wrong).

@searls

This comment has been minimized.

Show comment
Hide comment
@searls

searls Jan 12, 2016

Hey @myronmarston / @samphippen, how exactly does one git-bisect rspec-core without bundler failing to find prerelease versions of other rspec gems like rspec-support?

For instance, the very first bisect fails between the two release tags, because running bundle from the rspec-given project can't resolve rspec-support@3.4.0.pre

Bundler could not find compatible versions for gem "rspec-support":
  In Gemfile:
    rspec-core (>= 0) ruby depends on
      rspec-support (= 3.4.0.pre) ruby
Could not find gem 'rspec-support (= 3.4.0.pre) ruby', which is required by gem 'rspec-core (>= 0) ruby', in any of the sources.
Bundler could not find compatible versions for gem "rspec-support":
  In Gemfile:
    rspec-core (>= 0) ruby depends on
      rspec-support (= 3.4.0.pre) ruby
Could not find gem 'rspec-support (= 3.4.0.pre) ruby', which is required by gem 'rspec-core (>= 0) ruby', in any of the sources.

searls commented Jan 12, 2016

Hey @myronmarston / @samphippen, how exactly does one git-bisect rspec-core without bundler failing to find prerelease versions of other rspec gems like rspec-support?

For instance, the very first bisect fails between the two release tags, because running bundle from the rspec-given project can't resolve rspec-support@3.4.0.pre

Bundler could not find compatible versions for gem "rspec-support":
  In Gemfile:
    rspec-core (>= 0) ruby depends on
      rspec-support (= 3.4.0.pre) ruby
Could not find gem 'rspec-support (= 3.4.0.pre) ruby', which is required by gem 'rspec-core (>= 0) ruby', in any of the sources.
Bundler could not find compatible versions for gem "rspec-support":
  In Gemfile:
    rspec-core (>= 0) ruby depends on
      rspec-support (= 3.4.0.pre) ruby
Could not find gem 'rspec-support (= 3.4.0.pre) ruby', which is required by gem 'rspec-core (>= 0) ruby', in any of the sources.
@samphippen

This comment has been minimized.

Show comment
Hide comment
@samphippen

samphippen Jan 12, 2016

Hi @searls

So here's what I'd suggest you do

  1. Set up rspec-dev in your code folder

  2. Temporarily modify your dev Gemfile to point at each of the RSpec gems individually by path

    1. it'd look something like this
    source 'https://rubygems.org'
    
    gem 'rake'
    #gem 'rspec',    ENV['RSPEC_VERSION'] || '>= 2.14.0'
    gem 'minitest', '>= 4.3'
    gem 'sorcerer', '>= 0.3.7'
    
    gem "rspec-core", :path => "/Users/sam/dev/rspec/rspec-dev/repos/rspec-core"
    gem "rspec-expectations", :path => "/Users/sam/dev/rspec/rspec-dev/repos/rspec-expectations"
    gem "rspec-mocks", :path => "/Users/sam/dev/rspec/rspec-dev/repos/rspec-mocks"
    gem "rspec-support", :path => "/Users/sam/dev/rspec/rspec-dev/repos/rspec-support"
  3. start the bisection in RSpec core

  4. whenever you get an error, checkout the appropriate tag in each of the other repos, for example if you wanted to check out the first commit before 3.4.0 you'd do git checkout v3.4.0^.

You may have to repeat step 4 several times if you're bisecting over many versions. It's worth noting rspec-dev has a rake task to checkout all repositories to a specific version (bundle exec rake git:checkout[verison]) but you probably don't want that, as it'll interrupt your bisect.

samphippen commented Jan 12, 2016

Hi @searls

So here's what I'd suggest you do

  1. Set up rspec-dev in your code folder

  2. Temporarily modify your dev Gemfile to point at each of the RSpec gems individually by path

    1. it'd look something like this
    source 'https://rubygems.org'
    
    gem 'rake'
    #gem 'rspec',    ENV['RSPEC_VERSION'] || '>= 2.14.0'
    gem 'minitest', '>= 4.3'
    gem 'sorcerer', '>= 0.3.7'
    
    gem "rspec-core", :path => "/Users/sam/dev/rspec/rspec-dev/repos/rspec-core"
    gem "rspec-expectations", :path => "/Users/sam/dev/rspec/rspec-dev/repos/rspec-expectations"
    gem "rspec-mocks", :path => "/Users/sam/dev/rspec/rspec-dev/repos/rspec-mocks"
    gem "rspec-support", :path => "/Users/sam/dev/rspec/rspec-dev/repos/rspec-support"
  3. start the bisection in RSpec core

  4. whenever you get an error, checkout the appropriate tag in each of the other repos, for example if you wanted to check out the first commit before 3.4.0 you'd do git checkout v3.4.0^.

You may have to repeat step 4 several times if you're bisecting over many versions. It's worth noting rspec-dev has a rake task to checkout all repositories to a specific version (bundle exec rake git:checkout[verison]) but you probably don't want that, as it'll interrupt your bisect.

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Jan 12, 2016

Sam's suggestion should work but I think there are easier ways. I'm on my phone and don't want to type them out now but will follow up later.

myronmarston commented Jan 12, 2016

Sam's suggestion should work but I think there are easier ways. I'm on my phone and don't want to type them out now but will follow up later.

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Jan 12, 2016

I can see two ways to do this...

Since you are aren't trying to bisect across multiple RSpec minor versions, and instead are just trying to bisect all changes that occurred after 3.3.0 was released and before 3.4.0, you should be able to run the bisect from rspec/rspec-core@aaf2964 (the commit that bumped rspec-core to 3.4.0.pre -- just after 3.3.0 was released, and before any "real" changes were made) to rspec/rspec-core@0ab632a (the commit before 3.4.0.pre was bumped to 3.4.0, and after any "real" changes were made). In the other rspec libs, you'll just have to pin them to a commit where the version is 3.4.0.pre, like so:

gem 'rspec-core', path: "/path/to/your/git/clone"
gem 'rspec-expectations', github: "rspec/rspec-expectations", ref: "00315bebcf79d9bc40b22928c39e5791c8aaa923"
gem 'rspec-mocks', github: "rspec/rspec-mocks", ref: "ad03a31ab61a845550684e5de0274b2568dc2a9c"
gem 'rspec-support', github: "rspec/rspec-support", ref: "30b4a01dd039063ce9593e2094062c6f8a0eda34"

The commits I have listed above are the last commit in each repo where the version was 3.4.0.pre.

The other way would be to remove rubygems/bundler from loading RSpec. The version numbers aren't used at runtime by RSpec at all, and if you just add the lib dir for each gem to your $LOAD_PATH, and (temporarily) remove rspec from being listed in your Gemfile, you won't get version conflict errors (although you might get interface mismatch errors if you bisect to a point where the expected interfaces don't match -- but that's unlikely).

Let me know if you need more help, @searls.

myronmarston commented Jan 12, 2016

I can see two ways to do this...

Since you are aren't trying to bisect across multiple RSpec minor versions, and instead are just trying to bisect all changes that occurred after 3.3.0 was released and before 3.4.0, you should be able to run the bisect from rspec/rspec-core@aaf2964 (the commit that bumped rspec-core to 3.4.0.pre -- just after 3.3.0 was released, and before any "real" changes were made) to rspec/rspec-core@0ab632a (the commit before 3.4.0.pre was bumped to 3.4.0, and after any "real" changes were made). In the other rspec libs, you'll just have to pin them to a commit where the version is 3.4.0.pre, like so:

gem 'rspec-core', path: "/path/to/your/git/clone"
gem 'rspec-expectations', github: "rspec/rspec-expectations", ref: "00315bebcf79d9bc40b22928c39e5791c8aaa923"
gem 'rspec-mocks', github: "rspec/rspec-mocks", ref: "ad03a31ab61a845550684e5de0274b2568dc2a9c"
gem 'rspec-support', github: "rspec/rspec-support", ref: "30b4a01dd039063ce9593e2094062c6f8a0eda34"

The commits I have listed above are the last commit in each repo where the version was 3.4.0.pre.

The other way would be to remove rubygems/bundler from loading RSpec. The version numbers aren't used at runtime by RSpec at all, and if you just add the lib dir for each gem to your $LOAD_PATH, and (temporarily) remove rspec from being listed in your Gemfile, you won't get version conflict errors (although you might get interface mismatch errors if you bisect to a point where the expected interfaces don't match -- but that's unlikely).

Let me know if you need more help, @searls.

@searls

This comment has been minimized.

Show comment
Hide comment
@searls

searls Jan 12, 2016

I think I've gotten @samphippen's prescription working, however in order for require 'rspec' in my project to not blow up, I also had to have gem 'rspec', :path => '../../rspec/rspec' set up and checked out to v3.4.0^

searls commented Jan 12, 2016

I think I've gotten @samphippen's prescription working, however in order for require 'rspec' in my project to not blow up, I also had to have gem 'rspec', :path => '../../rspec/rspec' set up and checked out to v3.4.0^

@searls

This comment has been minimized.

Show comment
Hide comment
@searls

searls Jan 12, 2016

Hey @myronmarston I finally got a bisect running and it landed at rspec/rspec-core@aadd339

I'm investigating now

searls commented Jan 12, 2016

Hey @myronmarston I finally got a bisect running and it landed at rspec/rspec-core@aadd339

I'm investigating now

@searls

This comment has been minimized.

Show comment
Hide comment
@searls

searls Jan 12, 2016

Okay, below will look very confusing because the failing test is a test of RSpec's own output using RSpec, so each failure includes a full expected RSpec failure, with the failing assertion being to look for a regex match (as @jkowens fixed #19 )

On/after rspec/rspec-core@aadd339

Failures:

  1) Failing Messages when returning false from ToBool     Then { ios.out =~ /Failure\/Error: Then \{ ToBool.new \}/ }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing_messages_spec.rb:50
       expected: "F

Failures:

  1) Then with nesting   Then { ToBool.new }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/to_bool_returns_false.rb:11
         #<ToBool:0x007f9eb99e0ed0>
                  <- ToBool.new
         ToBool   <- ToBool
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/to_bool_returns_false.rb:11:in `block in Then'

Finished in 0.01053 seconds (files took 0.12305 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/to_bool_returns_false.rb:11 # Then with nesting   Then { ToBool.new }


"
       to match: /Failure\/Error: Then \{ ToBool.new \}/
         nil   <- ios.out =~ /Failure\/Error: Then \{ ToBool.new \}/
         "F

Failures:

  1) Then with nesting   Then { ToBool.new }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/to_bool_returns_false.rb:11
         #<ToBool:0x007f9eb99e0ed0>
                  <- ToBool.new
         ToBool   <- ToBool
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/to_bool_returns_false.rb:11:in `block in Then'

Finished in 0.01053 seconds (files took 0.12305 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/to_bool_returns_false.rb:11 # Then with nesting   Then { ToBool.new }


"
               <- ios.out
         #<struct IOS out="F

Failures:

  1) Then with nesting   Then { ToBool.new }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/to_bool_returns_false.rb:11
         #<ToBool:0x007f9eb99e0ed0>
                  <- ToBool.new
         ToBool   <- ToBool
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/to_bool_returns_false.rb:11:in `block in Then'

Finished in 0.01053 seconds (files took 0.12305 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/to_bool_returns_false.rb:11 # Then with nesting   Then { ToBool.new }


", err="">
               <- ios
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing_messages_spec.rb:50:in `block in Then'

  2) Failing Messages with an oddly formatted then     Then { ios.out =~ /Failure\/Error: Then \{ result == \['a',$/ }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing_messages_spec.rb:55
       expected: "F

Failures:

  1) arrays split over multiple lines   Then { result == ['a',
                    'a'] }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5
       expected: \"anything\"
       to equal: [\"a\", \"a\"]
         false        <- result == [\"a\", \"a\"]
         \"anything\"   <- result
         [\"a\", \"a\"]   <- [\"a\", \"a\"]
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'

Finished in 0.01085 seconds (files took 0.12495 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',
                    'a'] }


"
       to match: /Failure\/Error: Then \{ result == \['a',$/
         nil   <- ios.out =~ /Failure\/Error: Then \{ result == \['a',$/
         "F

Failures:

  1) arrays split over multiple lines   Then { result == ['a',
                    'a'] }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5
       expected: \"anything\"
       to equal: [\"a\", \"a\"]
         false        <- result == [\"a\", \"a\"]
         \"anything\"   <- result
         [\"a\", \"a\"]   <- [\"a\", \"a\"]
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'

Finished in 0.01085 seconds (files took 0.12495 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',
                    'a'] }


"
               <- ios.out
         #<struct IOS out="F

Failures:

  1) arrays split over multiple lines   Then { result == ['a',
                    'a'] }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5
       expected: \"anything\"
       to equal: [\"a\", \"a\"]
         false        <- result == [\"a\", \"a\"]
         \"anything\"   <- result
         [\"a\", \"a\"]   <- [\"a\", \"a\"]
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'

Finished in 0.01085 seconds (files took 0.12495 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',
                    'a'] }


", err="">
               <- ios
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing_messages_spec.rb:55:in `block in Then'

Finished in 1.27 seconds (files took 0.15754 seconds to load)
38 examples, 2 failures

Failed examples:

rspec ./examples/integration/failing_messages_spec.rb:50 # Failing Messages when returning false from ToBool     Then { ios.out =~ /Failure\/Error: Then \{ ToBool.new \}/ }

rspec ./examples/integration/failing_messages_spec.rb:55 # Failing Messages with an oddly formatted then     Then { ios.out =~ /Failure\/Error: Then \{ result == \['a',$/ }

Prior to rspec/rspec-core@aadd339

Next, I intentionally broke the two tests on the last good commit (parent to rspec/rspec-core@aadd339) to see what the failure message looks like when the rspec-given test is happy:

Failures:

  1) Failing Messages when returning false from ToBool     Then { ios.out =~ /JKLOL1-Failure\/Error: Then \{ ToBool.new \}/ }

     Failure/Error: Then { ios.out =~ /JKLOL1-Failure\/Error: Then \{ ToBool.new \}/ }
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing_messages_spec.rb:50
       expected: "F

Failures:

  1) Then with nesting   Then { ToBool.new }

     Failure/Error: Then { ToBool.new }
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/to_bool_returns_false.rb:11
         #<ToBool:0x007ff783039ee0>
                  <- ToBool.new
         ToBool   <- ToBool
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/to_bool_returns_false.rb:11:in `block in Then'

Finished in 0.00822 seconds (files took 0.08627 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/to_bool_returns_false.rb:11 # Then with nesting   Then { ToBool.new }


"
       to match: /JKLOL1-Failure\/Error: Then \{ ToBool.new \}/
         nil   <- ios.out =~ /JKLOL1-Failure\/Error: Then \{ ToBool.new \}/
         "F

Failures:

  1) Then with nesting   Then { ToBool.new }

     Failure/Error: Then { ToBool.new }
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/to_bool_returns_false.rb:11
         #<ToBool:0x007ff783039ee0>
                  <- ToBool.new
         ToBool   <- ToBool
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/to_bool_returns_false.rb:11:in `block in Then'

Finished in 0.00822 seconds (files took 0.08627 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/to_bool_returns_false.rb:11 # Then with nesting   Then { ToBool.new }


"
               <- ios.out
         #<struct IOS out="F

Failures:

  1) Then with nesting   Then { ToBool.new }

     Failure/Error: Then { ToBool.new }
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/to_bool_returns_false.rb:11
         #<ToBool:0x007ff783039ee0>
                  <- ToBool.new
         ToBool   <- ToBool
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/to_bool_returns_false.rb:11:in `block in Then'

Finished in 0.00822 seconds (files took 0.08627 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/to_bool_returns_false.rb:11 # Then with nesting   Then { ToBool.new }


", err="">
               <- ios
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing_messages_spec.rb:50:in `block in Then'

  2) Failing Messages with an oddly formatted then     Then { ios.out =~ /JKLOL2-Failure\/Error: Then \{ result == \['a',$/ }

     Failure/Error: Then { ios.out =~ /JKLOL2-Failure\/Error: Then \{ result == \['a',$/ }
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing_messages_spec.rb:55
       expected: "F

Failures:

  1) arrays split over multiple lines   Then { result == ['a',
                    'a'] }

     Failure/Error: Then { result == ['a',
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5
       expected: \"anything\"
       to equal: [\"a\", \"a\"]
         false        <- result == [\"a\", \"a\"]
         \"anything\"   <- result
         [\"a\", \"a\"]   <- [\"a\", \"a\"]
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'

Finished in 0.00768 seconds (files took 0.08796 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',
                    'a'] }


"
       to match: /JKLOL2-Failure\/Error: Then \{ result == \['a',$/
         nil   <- ios.out =~ /JKLOL2-Failure\/Error: Then \{ result == \['a',$/
         "F

Failures:

  1) arrays split over multiple lines   Then { result == ['a',
                    'a'] }

     Failure/Error: Then { result == ['a',
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5
       expected: \"anything\"
       to equal: [\"a\", \"a\"]
         false        <- result == [\"a\", \"a\"]
         \"anything\"   <- result
         [\"a\", \"a\"]   <- [\"a\", \"a\"]
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'

Finished in 0.00768 seconds (files took 0.08796 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',
                    'a'] }


"
               <- ios.out
         #<struct IOS out="F

Failures:

  1) arrays split over multiple lines   Then { result == ['a',
                    'a'] }

     Failure/Error: Then { result == ['a',
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5
       expected: \"anything\"
       to equal: [\"a\", \"a\"]
         false        <- result == [\"a\", \"a\"]
         \"anything\"   <- result
         [\"a\", \"a\"]   <- [\"a\", \"a\"]
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'

Finished in 0.00768 seconds (files took 0.08796 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',
                    'a'] }


", err="">
               <- ios
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing_messages_spec.rb:55:in `block in Then'

Finished in 2.02 seconds (files took 0.21063 seconds to load)
38 examples, 2 failures

Failed examples:

rspec ./examples/integration/failing_messages_spec.rb:50 # Failing Messages when returning false from ToBool     Then { ios.out =~ /JKLOL1-Failure\/Error: Then \{ ToBool.new \}/ }

rspec ./examples/integration/failing_messages_spec.rb:55 # Failing Messages with an oddly formatted then     Then { ios.out =~ /JKLOL2-Failure\/Error: Then \{ result == \['a',$/ }

searls commented Jan 12, 2016

Okay, below will look very confusing because the failing test is a test of RSpec's own output using RSpec, so each failure includes a full expected RSpec failure, with the failing assertion being to look for a regex match (as @jkowens fixed #19 )

On/after rspec/rspec-core@aadd339

Failures:

  1) Failing Messages when returning false from ToBool     Then { ios.out =~ /Failure\/Error: Then \{ ToBool.new \}/ }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing_messages_spec.rb:50
       expected: "F

Failures:

  1) Then with nesting   Then { ToBool.new }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/to_bool_returns_false.rb:11
         #<ToBool:0x007f9eb99e0ed0>
                  <- ToBool.new
         ToBool   <- ToBool
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/to_bool_returns_false.rb:11:in `block in Then'

Finished in 0.01053 seconds (files took 0.12305 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/to_bool_returns_false.rb:11 # Then with nesting   Then { ToBool.new }


"
       to match: /Failure\/Error: Then \{ ToBool.new \}/
         nil   <- ios.out =~ /Failure\/Error: Then \{ ToBool.new \}/
         "F

Failures:

  1) Then with nesting   Then { ToBool.new }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/to_bool_returns_false.rb:11
         #<ToBool:0x007f9eb99e0ed0>
                  <- ToBool.new
         ToBool   <- ToBool
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/to_bool_returns_false.rb:11:in `block in Then'

Finished in 0.01053 seconds (files took 0.12305 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/to_bool_returns_false.rb:11 # Then with nesting   Then { ToBool.new }


"
               <- ios.out
         #<struct IOS out="F

Failures:

  1) Then with nesting   Then { ToBool.new }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/to_bool_returns_false.rb:11
         #<ToBool:0x007f9eb99e0ed0>
                  <- ToBool.new
         ToBool   <- ToBool
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/to_bool_returns_false.rb:11:in `block in Then'

Finished in 0.01053 seconds (files took 0.12305 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/to_bool_returns_false.rb:11 # Then with nesting   Then { ToBool.new }


", err="">
               <- ios
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing_messages_spec.rb:50:in `block in Then'

  2) Failing Messages with an oddly formatted then     Then { ios.out =~ /Failure\/Error: Then \{ result == \['a',$/ }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing_messages_spec.rb:55
       expected: "F

Failures:

  1) arrays split over multiple lines   Then { result == ['a',
                    'a'] }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5
       expected: \"anything\"
       to equal: [\"a\", \"a\"]
         false        <- result == [\"a\", \"a\"]
         \"anything\"   <- result
         [\"a\", \"a\"]   <- [\"a\", \"a\"]
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'

Finished in 0.01085 seconds (files took 0.12495 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',
                    'a'] }


"
       to match: /Failure\/Error: Then \{ result == \['a',$/
         nil   <- ios.out =~ /Failure\/Error: Then \{ result == \['a',$/
         "F

Failures:

  1) arrays split over multiple lines   Then { result == ['a',
                    'a'] }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5
       expected: \"anything\"
       to equal: [\"a\", \"a\"]
         false        <- result == [\"a\", \"a\"]
         \"anything\"   <- result
         [\"a\", \"a\"]   <- [\"a\", \"a\"]
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'

Finished in 0.01085 seconds (files took 0.12495 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',
                    'a'] }


"
               <- ios.out
         #<struct IOS out="F

Failures:

  1) arrays split over multiple lines   Then { result == ['a',
                    'a'] }

     Failure/Error: ::RSpec::Expectations.fail_with(*args)

       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5
       expected: \"anything\"
       to equal: [\"a\", \"a\"]
         false        <- result == [\"a\", \"a\"]
         \"anything\"   <- result
         [\"a\", \"a\"]   <- [\"a\", \"a\"]
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'

Finished in 0.01085 seconds (files took 0.12495 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',
                    'a'] }


", err="">
               <- ios
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing_messages_spec.rb:55:in `block in Then'

Finished in 1.27 seconds (files took 0.15754 seconds to load)
38 examples, 2 failures

Failed examples:

rspec ./examples/integration/failing_messages_spec.rb:50 # Failing Messages when returning false from ToBool     Then { ios.out =~ /Failure\/Error: Then \{ ToBool.new \}/ }

rspec ./examples/integration/failing_messages_spec.rb:55 # Failing Messages with an oddly formatted then     Then { ios.out =~ /Failure\/Error: Then \{ result == \['a',$/ }

Prior to rspec/rspec-core@aadd339

Next, I intentionally broke the two tests on the last good commit (parent to rspec/rspec-core@aadd339) to see what the failure message looks like when the rspec-given test is happy:

Failures:

  1) Failing Messages when returning false from ToBool     Then { ios.out =~ /JKLOL1-Failure\/Error: Then \{ ToBool.new \}/ }

     Failure/Error: Then { ios.out =~ /JKLOL1-Failure\/Error: Then \{ ToBool.new \}/ }
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing_messages_spec.rb:50
       expected: "F

Failures:

  1) Then with nesting   Then { ToBool.new }

     Failure/Error: Then { ToBool.new }
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/to_bool_returns_false.rb:11
         #<ToBool:0x007ff783039ee0>
                  <- ToBool.new
         ToBool   <- ToBool
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/to_bool_returns_false.rb:11:in `block in Then'

Finished in 0.00822 seconds (files took 0.08627 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/to_bool_returns_false.rb:11 # Then with nesting   Then { ToBool.new }


"
       to match: /JKLOL1-Failure\/Error: Then \{ ToBool.new \}/
         nil   <- ios.out =~ /JKLOL1-Failure\/Error: Then \{ ToBool.new \}/
         "F

Failures:

  1) Then with nesting   Then { ToBool.new }

     Failure/Error: Then { ToBool.new }
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/to_bool_returns_false.rb:11
         #<ToBool:0x007ff783039ee0>
                  <- ToBool.new
         ToBool   <- ToBool
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/to_bool_returns_false.rb:11:in `block in Then'

Finished in 0.00822 seconds (files took 0.08627 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/to_bool_returns_false.rb:11 # Then with nesting   Then { ToBool.new }


"
               <- ios.out
         #<struct IOS out="F

Failures:

  1) Then with nesting   Then { ToBool.new }

     Failure/Error: Then { ToBool.new }
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/to_bool_returns_false.rb:11
         #<ToBool:0x007ff783039ee0>
                  <- ToBool.new
         ToBool   <- ToBool
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/to_bool_returns_false.rb:11:in `block in Then'

Finished in 0.00822 seconds (files took 0.08627 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/to_bool_returns_false.rb:11 # Then with nesting   Then { ToBool.new }


", err="">
               <- ios
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing_messages_spec.rb:50:in `block in Then'

  2) Failing Messages with an oddly formatted then     Then { ios.out =~ /JKLOL2-Failure\/Error: Then \{ result == \['a',$/ }

     Failure/Error: Then { ios.out =~ /JKLOL2-Failure\/Error: Then \{ result == \['a',$/ }
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing_messages_spec.rb:55
       expected: "F

Failures:

  1) arrays split over multiple lines   Then { result == ['a',
                    'a'] }

     Failure/Error: Then { result == ['a',
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5
       expected: \"anything\"
       to equal: [\"a\", \"a\"]
         false        <- result == [\"a\", \"a\"]
         \"anything\"   <- result
         [\"a\", \"a\"]   <- [\"a\", \"a\"]
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'

Finished in 0.00768 seconds (files took 0.08796 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',
                    'a'] }


"
       to match: /JKLOL2-Failure\/Error: Then \{ result == \['a',$/
         nil   <- ios.out =~ /JKLOL2-Failure\/Error: Then \{ result == \['a',$/
         "F

Failures:

  1) arrays split over multiple lines   Then { result == ['a',
                    'a'] }

     Failure/Error: Then { result == ['a',
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5
       expected: \"anything\"
       to equal: [\"a\", \"a\"]
         false        <- result == [\"a\", \"a\"]
         \"anything\"   <- result
         [\"a\", \"a\"]   <- [\"a\", \"a\"]
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'

Finished in 0.00768 seconds (files took 0.08796 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',
                    'a'] }


"
               <- ios.out
         #<struct IOS out="F

Failures:

  1) arrays split over multiple lines   Then { result == ['a',
                    'a'] }

     Failure/Error: Then { result == ['a',
       Then expression failed at /Users/justin/code/rspec-given/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5
       expected: \"anything\"
       to equal: [\"a\", \"a\"]
         false        <- result == [\"a\", \"a\"]
         \"anything\"   <- result
         [\"a\", \"a\"]   <- [\"a\", \"a\"]
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'

Finished in 0.00768 seconds (files took 0.08796 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',
                    'a'] }


", err="">
               <- ios
     # ./lib/given/rspec/framework.rb:24:in `fail_with'
     # ./lib/given/module_methods.rb:59:in `fail_with'
     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'
     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'
     # ./lib/given/extensions.rb:102:in `_gvn_then'
     # ./examples/integration/failing_messages_spec.rb:55:in `block in Then'

Finished in 2.02 seconds (files took 0.21063 seconds to load)
38 examples, 2 failures

Failed examples:

rspec ./examples/integration/failing_messages_spec.rb:50 # Failing Messages when returning false from ToBool     Then { ios.out =~ /JKLOL1-Failure\/Error: Then \{ ToBool.new \}/ }

rspec ./examples/integration/failing_messages_spec.rb:55 # Failing Messages with an oddly formatted then     Then { ios.out =~ /JKLOL2-Failure\/Error: Then \{ result == \['a',$/ }
@searls

This comment has been minimized.

Show comment
Hide comment
@searls

searls Jan 12, 2016

So fundamentally, it looks like rspec/rspec-core@aadd339 changed the expected behavior of how RSpec prints rspec-given failures from this:

Failure/Error: Then { ToBool.new }

to this:

Failure/Error: ::RSpec::Expectations.fail_with(*args)

Obviously, Jim designed rspec-given to print the former, and it's much more useful to the end-user than the latter, so I do consider this to be broken from our perspective (though not necessarily a thing that should be fixed in rspec; I don't know yet). However, it does mean that I don't think it'd be prudent to simply shut up the failure by merging #19, either, so I'll be closing it (sorry @jkowens!).

Can you provide any additional wisdom @myronmarston while I dig into the change from the rspec-core side?

searls commented Jan 12, 2016

So fundamentally, it looks like rspec/rspec-core@aadd339 changed the expected behavior of how RSpec prints rspec-given failures from this:

Failure/Error: Then { ToBool.new }

to this:

Failure/Error: ::RSpec::Expectations.fail_with(*args)

Obviously, Jim designed rspec-given to print the former, and it's much more useful to the end-user than the latter, so I do consider this to be broken from our perspective (though not necessarily a thing that should be fixed in rspec; I don't know yet). However, it does mean that I don't think it'd be prudent to simply shut up the failure by merging #19, either, so I'll be closing it (sorry @jkowens!).

Can you provide any additional wisdom @myronmarston while I dig into the change from the rspec-core side?

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Jan 12, 2016

Thanks for digging in @searls. I just realized what's going on. The detail you provided above definitely helped.

Before the change in rspec-core, RSpec had a very simple rule for what line of code to look for: it looked for the first backtrace line that matched the file path of the spec file containing the failing example. We don't want to print the code from the first line of the backtrace because that would usually be a line in rspec-expectations where the expectation failure raised an error. This usually worked but not always. For example, if you defined a global before hook in spec/spec_helper.rb where the exception occurred, it would not find it and would print "Unable to find matching line from backtrace". If the failure occurred deep within your own lib code, it would print the call site in the spec where it called whatever eventually hit the exception.

In that change, RSpec became much smarter:

  • It now understands the concept of "project source directories" -- these are the dirs where your project's source code resides. It defaults to lib, spec and app.
  • RSpec now looks for the first line from one of those directories, and prints that line, which addresses the issues I mentioned above.
  • If it can't find any lines in those directories, it falls back to just printing the first line, because that's better than nothing.

This is affecting rspec-given because rspec-given is a test framework, with test-framework code in the lib directory when you run rspec-given's test suite. Specifically, this line is in your lib directory, and is the first line in one of the project_source_dirs that RSpec can find, so it prints it.

It should be noted that I do not expect this issue to affect rspec-given end-users at all, because when an rspec-given user uses rspec-given, they do not have rspec-given's source code in their lib directory. They have their own source code in that directory and the problem should not occur. But it occurs when you run rspec-given's test suite because the rspec-given source code is in the lib directory.

The solution is to change your RSpec configuration in your spec_helper.rb file so that it removes lib from project_source_dirs:

RSpec.configure do |config|
  config.project_source_dirs -= ["lib"]
end

Note that we had to do the same thing for rspec-expectations (rspec/rspec-expectations#871) and rspec-mocks (rspec/rspec-mocks#1036) and it makes sense that you have to do it here as well.

Sorry for the confusion, and not realizing the problem sooner!

myronmarston commented Jan 12, 2016

Thanks for digging in @searls. I just realized what's going on. The detail you provided above definitely helped.

Before the change in rspec-core, RSpec had a very simple rule for what line of code to look for: it looked for the first backtrace line that matched the file path of the spec file containing the failing example. We don't want to print the code from the first line of the backtrace because that would usually be a line in rspec-expectations where the expectation failure raised an error. This usually worked but not always. For example, if you defined a global before hook in spec/spec_helper.rb where the exception occurred, it would not find it and would print "Unable to find matching line from backtrace". If the failure occurred deep within your own lib code, it would print the call site in the spec where it called whatever eventually hit the exception.

In that change, RSpec became much smarter:

  • It now understands the concept of "project source directories" -- these are the dirs where your project's source code resides. It defaults to lib, spec and app.
  • RSpec now looks for the first line from one of those directories, and prints that line, which addresses the issues I mentioned above.
  • If it can't find any lines in those directories, it falls back to just printing the first line, because that's better than nothing.

This is affecting rspec-given because rspec-given is a test framework, with test-framework code in the lib directory when you run rspec-given's test suite. Specifically, this line is in your lib directory, and is the first line in one of the project_source_dirs that RSpec can find, so it prints it.

It should be noted that I do not expect this issue to affect rspec-given end-users at all, because when an rspec-given user uses rspec-given, they do not have rspec-given's source code in their lib directory. They have their own source code in that directory and the problem should not occur. But it occurs when you run rspec-given's test suite because the rspec-given source code is in the lib directory.

The solution is to change your RSpec configuration in your spec_helper.rb file so that it removes lib from project_source_dirs:

RSpec.configure do |config|
  config.project_source_dirs -= ["lib"]
end

Note that we had to do the same thing for rspec-expectations (rspec/rspec-expectations#871) and rspec-mocks (rspec/rspec-mocks#1036) and it makes sense that you have to do it here as well.

Sorry for the confusion, and not realizing the problem sooner!

@searls

This comment has been minimized.

Show comment
Hide comment
@searls

searls Jan 12, 2016

Hey @myronmarston, simply dropping that into either/both of the spec_helper or the examples/example_helper.rb did not have any effect.

searls commented Jan 12, 2016

Hey @myronmarston, simply dropping that into either/both of the spec_helper or the examples/example_helper.rb did not have any effect.

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Jan 12, 2016

Odd. I'll take a look. Since I've never worked on rspec-given or even looked at the code base...can you provide a minimal reproduction command to repro the problem locally? And is it happening on your master branch or just on this PR?

myronmarston commented Jan 12, 2016

Odd. I'll take a look. Since I've never worked on rspec-given or even looked at the code base...can you provide a minimal reproduction command to repro the problem locally? And is it happening on your master branch or just on this PR?

@searls

This comment has been minimized.

Show comment
Hide comment
@searls

searls Jan 12, 2016

It's happening constantly b/c of the lax version specifier on RSpec.

You should be able to replicate by just pulling down, running bundle and running bundle exec rspec ./examples/integration/failing_messages_spec.rb

searls commented Jan 12, 2016

It's happening constantly b/c of the lax version specifier on RSpec.

You should be able to replicate by just pulling down, running bundle and running bundle exec rspec ./examples/integration/failing_messages_spec.rb

@jkowens

This comment has been minimized.

Show comment
Hide comment
@jkowens

jkowens Jan 12, 2016

@searls I believe require 'example_helper' should replace require 'rspec/given' in each of the specs under the examples directory.

Once I did this, the number of failures was reduced from 2 to 1.

Failures:

  1) Failing Messages with an oddly formatted then     Then { ios.out =~ /Failure\/Error: Then \{ result == \['a',$/m }

     Failure/Error: Then { ios.out =~ /Failure\/Error: Then \{ result == \['a',$/m }

       Then expression failed at /Users/jordan/projects/rspec-given/examples/integration/failing_messages_spec.rb:55
       expected: "F\n\nFailures:\n\n  1) arrays split over multiple lines   Then { result == ['a',\n                    'a'] }\n\n     Failure/Error:\n       Then { result == ['a',\n                         'a'] }\n\n       Then expression failed at /Users/jordan/projects/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5\n       expected: \"anything\"\n       to equal: [\"a\", \"a\"]\n         false        <- result == [\"a\", \"a\"]\n         \"anything\"   <- result\n         [\"a\", \"a\"]   <- [\"a\", \"a\"]\n     # ./lib/given/rspec/framework.rb:24:in `fail_with'\n     # ./lib/given/module_methods.rb:59:in `fail_with'\n     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'\n     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'\n     # ./lib/given/extensions.rb:102:in `_gvn_then'\n     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'\n\nFinished in 0.01321 seconds (files took 0.1142 seconds to load)\n1 example, 1 failure\n\nFailed examples:\n\nrspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',\n                    'a'] }\n\n\n"
       to match: /Failure\/Error: Then \{ result == \['a',$/

The expected output is almost achieved, but there is a slight difference:

Expected:

Failure/Error: Then { result == ['a',\n

Actual:

Failure/Error:\n       Then { result == ['a',\n

jkowens commented Jan 12, 2016

@searls I believe require 'example_helper' should replace require 'rspec/given' in each of the specs under the examples directory.

Once I did this, the number of failures was reduced from 2 to 1.

Failures:

  1) Failing Messages with an oddly formatted then     Then { ios.out =~ /Failure\/Error: Then \{ result == \['a',$/m }

     Failure/Error: Then { ios.out =~ /Failure\/Error: Then \{ result == \['a',$/m }

       Then expression failed at /Users/jordan/projects/rspec-given/examples/integration/failing_messages_spec.rb:55
       expected: "F\n\nFailures:\n\n  1) arrays split over multiple lines   Then { result == ['a',\n                    'a'] }\n\n     Failure/Error:\n       Then { result == ['a',\n                         'a'] }\n\n       Then expression failed at /Users/jordan/projects/rspec-given/examples/integration/failing/oddly_formatted_then.rb:5\n       expected: \"anything\"\n       to equal: [\"a\", \"a\"]\n         false        <- result == [\"a\", \"a\"]\n         \"anything\"   <- result\n         [\"a\", \"a\"]   <- [\"a\", \"a\"]\n     # ./lib/given/rspec/framework.rb:24:in `fail_with'\n     # ./lib/given/module_methods.rb:59:in `fail_with'\n     # ./lib/given/extensions.rb:127:in `_gvn_naturally_assert'\n     # ./lib/given/extensions.rb:118:in `_gvn_evaluate'\n     # ./lib/given/extensions.rb:102:in `_gvn_then'\n     # ./examples/integration/failing/oddly_formatted_then.rb:5:in `block in Then'\n\nFinished in 0.01321 seconds (files took 0.1142 seconds to load)\n1 example, 1 failure\n\nFailed examples:\n\nrspec ./examples/integration/failing/oddly_formatted_then.rb:5 # arrays split over multiple lines   Then { result == ['a',\n                    'a'] }\n\n\n"
       to match: /Failure\/Error: Then \{ result == \['a',$/

The expected output is almost achieved, but there is a slight difference:

Expected:

Failure/Error: Then { result == ['a',\n

Actual:

Failure/Error:\n       Then { result == ['a',\n
@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Jan 12, 2016

PR incoming with a fix.

myronmarston commented Jan 12, 2016

PR incoming with a fix.

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston commented Jan 12, 2016

Fix is in #20.

@searls searls closed this in #20 Jan 13, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment