Add #passed? and #failed? methods to ExampleGroup

[natritmeyer]

I often want to diagnostic stuff in the after(:each) block of a failing test, but there is no way to determine the outcome of a test. As described here: https://github.com/cucumber/cucumber/wiki/Hooks, cucumber has the following construct:

After do |scenario|
  if scenario.failed?
    #do stuff
  end
end

I have put together a monkey patch which lets me do that in rspec. The gist is here:

https://gist.github.com/1009181

It provides #passed? and #failed? methods on RSpec::Core::ExampleGroup

after(:each) do
  if failed?
    #do stuff
  end
end

It would be great if rspec provided something like this.

[dchelimsky]

You can do this right now:

after(:each) do
  if example.execution_result[:status] == 'failed'
    # do stuff
  end
end

I'm open to a more direct API, but probably on the example itself. Something like:

after(:each) do
  if example.failed?
    # do stuff
  end
end

[natritmeyer]

On the example itself makes sense.

I just tried the following:

describe "something" do
  it "should pass" do
    1.should == 1
  end

  it "should fail" do
    1.should == 2
  end

  after(:each) do
    puts "FAILED!!!" if example.execution_result[:status] == 'failed'
  end
end

...but I didn't see the "FAILED!!!" that I was expecting to see in the output. Here's what I got:

nat$ rspec thing2.rb 
.F

Failures:

  1) something should fail
     Failure/Error: 1.should == 2
       expected: 2
            got: 1 (using ==)
     # ./thing2.rb:7:in `block (2 levels) in <top (required)>'

Finished in 0.00103 seconds
2 examples, 1 failure

What am I missing?

[natritmeyer]

Playing with execution_result[:status], when I replace the after block of the above example with the following...

  after(:each) do
    p example.execution_result[:status]
  end

...all I see in the output is 'nil's:

nat$ rspec thing2.rb 
nil
.nil
F

That's not right...

[dchelimsky]

@natritmeyer - I was wrong about the timing - sorry for sending you in the wrong direction. The execution_result isn't populated until after the after hooks, which makes sense because an exception raised in an after hook triggers a failure. What you can check for now as a workaround is the @exception instance variable. This is clearly invasive and not intended to be a long-term solution for you, but ...

after(:each) do
  if example.instance_variable_get("@exception")
    # do something
  end
end

[natritmeyer]

@dchelimsky That's exactly what the aforementioned monkeypatch does. Thanks! For reference, here it is:

class RSpec::Core::ExampleGroup 
  def passed?
    example.instance_variable_get(:@exception).nil? 
  end

  def failed? 
    !passed? 
  end
end

[dchelimsky]

Again, the example isn't really finished until after all the after hooks have run, so trying to hook this in an after block is unreliable, even if it's a method on the example or example_group.

After a bit more thought, one mechanism that is supported right now is custom formatters. You can use an arbitrary number of formatters, so you are free to do something like this:

class FailureReporter
  def example_failed(example)
    p "#{example.full_description} failed: #{example.execution_result[:exception].message}"
  end

  def method_missing(m, *a, &b)
    # ignore other messages
  end
end

RSpec.configure do |c|
  c.add_formatter FailureReporter
end

describe "something" do
  it "does something" do
    raise "oops, something went wrong"
  end
end

$ rspec example_spec.rb -fd

something
  does something (FAILED - 1)
"something does something failed: oops, something went wrong"

Does that satisfy your need?

[natritmeyer]

An interesting idea, but a formatter will apply to all the scenarios being executed, not just one describe block. Also, I'm not sure if instance variables from scenarios are available to formatters (what I really want to do is to write out an instance variable on scenario failure in only one spec.rb file).

So an example isn't finished until the after block is finished. But... if an exception occured in the body of the scenario then the the scenario has failed and nothing that occurs in the after block will change that. So though it may not be possible to know if a scenario has passed, it is possible to know that it has failed, right? If so, it would not be unreasonable to have a #failed? method available in the after block (whether on the example or example group)...right?

[dchelimsky]

Also, I'm not sure if instance variables from scenarios are available to formatters

Instance variables are not available to formatters.

So though it may not be possible to know if a scenario has passed, it is possible to know that it has failed, right?

But then you'd get a false negative if the failure occurs in a subsequent after block. That might be OK for your situation, but it's not OK for a general purpose addition to the core API.

What if we add an accessor for the @exception instance variable? This wouldn't give you the nice pass/fail method names, but at least you'd be able to check to see if there was an exception more gracefully:

after(:each) do
  if example.exception
    # do something
  end
end

WDYT?

[natritmeyer]

Interesting. Thinking about the false-fail aspect of it, I guess the failed? method is conceptually closer to failed_yet?.

Exposing the exception would work. You're right - it's not as pretty as failed?, but it would do the trick.