New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add #passed? and #failed? methods to ExampleGroup #401

Closed
natritmeyer opened this Issue Jun 5, 2011 · 9 comments

Comments

Projects
None yet
3 participants
@natritmeyer

natritmeyer commented Jun 5, 2011

I often want to diagnostic stuff in the after(:each) block of a failing test, but there is no way to determine the outcome of a test. As described here: https://github.com/cucumber/cucumber/wiki/Hooks, cucumber has the following construct:

After do |scenario|
  if scenario.failed?
    #do stuff
  end
end

I have put together a monkey patch which lets me do that in rspec. The gist is here:

https://gist.github.com/1009181

It provides #passed? and #failed? methods on RSpec::Core::ExampleGroup

after(:each) do
  if failed?
    #do stuff
  end
end

It would be great if rspec provided something like this.

@dchelimsky

This comment has been minimized.

Show comment
Hide comment
@dchelimsky

dchelimsky Jun 5, 2011

Member

You can do this right now:

after(:each) do
  if example.execution_result[:status] == 'failed'
    # do stuff
  end
end

I'm open to a more direct API, but probably on the example itself. Something like:

after(:each) do
  if example.failed?
    # do stuff
  end
end
Member

dchelimsky commented Jun 5, 2011

You can do this right now:

after(:each) do
  if example.execution_result[:status] == 'failed'
    # do stuff
  end
end

I'm open to a more direct API, but probably on the example itself. Something like:

after(:each) do
  if example.failed?
    # do stuff
  end
end
@natritmeyer

This comment has been minimized.

Show comment
Hide comment
@natritmeyer

natritmeyer Jun 5, 2011

On the example itself makes sense.

I just tried the following:

describe "something" do
  it "should pass" do
    1.should == 1
  end

  it "should fail" do
    1.should == 2
  end

  after(:each) do
    puts "FAILED!!!" if example.execution_result[:status] == 'failed'
  end
end

...but I didn't see the "FAILED!!!" that I was expecting to see in the output. Here's what I got:

nat$ rspec thing2.rb 
.F

Failures:

  1) something should fail
     Failure/Error: 1.should == 2
       expected: 2
            got: 1 (using ==)
     # ./thing2.rb:7:in `block (2 levels) in <top (required)>'

Finished in 0.00103 seconds
2 examples, 1 failure

What am I missing?

natritmeyer commented Jun 5, 2011

On the example itself makes sense.

I just tried the following:

describe "something" do
  it "should pass" do
    1.should == 1
  end

  it "should fail" do
    1.should == 2
  end

  after(:each) do
    puts "FAILED!!!" if example.execution_result[:status] == 'failed'
  end
end

...but I didn't see the "FAILED!!!" that I was expecting to see in the output. Here's what I got:

nat$ rspec thing2.rb 
.F

Failures:

  1) something should fail
     Failure/Error: 1.should == 2
       expected: 2
            got: 1 (using ==)
     # ./thing2.rb:7:in `block (2 levels) in <top (required)>'

Finished in 0.00103 seconds
2 examples, 1 failure

What am I missing?

@natritmeyer

This comment has been minimized.

Show comment
Hide comment
@natritmeyer

natritmeyer Jun 5, 2011

Playing with execution_result[:status], when I replace the after block of the above example with the following...

  after(:each) do
    p example.execution_result[:status]
  end

...all I see in the output is 'nil's:

nat$ rspec thing2.rb 
nil
.nil
F

That's not right...

natritmeyer commented Jun 5, 2011

Playing with execution_result[:status], when I replace the after block of the above example with the following...

  after(:each) do
    p example.execution_result[:status]
  end

...all I see in the output is 'nil's:

nat$ rspec thing2.rb 
nil
.nil
F

That's not right...

@dchelimsky

This comment has been minimized.

Show comment
Hide comment
@dchelimsky

dchelimsky Jun 5, 2011

Member

@natritmeyer - I was wrong about the timing - sorry for sending you in the wrong direction. The execution_result isn't populated until after the after hooks, which makes sense because an exception raised in an after hook triggers a failure. What you can check for now as a workaround is the @exception instance variable. This is clearly invasive and not intended to be a long-term solution for you, but ...

after(:each) do
  if example.instance_variable_get("@exception")
    # do something
  end
end
Member

dchelimsky commented Jun 5, 2011

@natritmeyer - I was wrong about the timing - sorry for sending you in the wrong direction. The execution_result isn't populated until after the after hooks, which makes sense because an exception raised in an after hook triggers a failure. What you can check for now as a workaround is the @exception instance variable. This is clearly invasive and not intended to be a long-term solution for you, but ...

after(:each) do
  if example.instance_variable_get("@exception")
    # do something
  end
end
@natritmeyer

This comment has been minimized.

Show comment
Hide comment
@natritmeyer

natritmeyer Jun 5, 2011

@dchelimsky That's exactly what the aforementioned monkeypatch does. Thanks! For reference, here it is:

class RSpec::Core::ExampleGroup 
  def passed?
    example.instance_variable_get(:@exception).nil? 
  end

  def failed? 
    !passed? 
  end
end

natritmeyer commented Jun 5, 2011

@dchelimsky That's exactly what the aforementioned monkeypatch does. Thanks! For reference, here it is:

class RSpec::Core::ExampleGroup 
  def passed?
    example.instance_variable_get(:@exception).nil? 
  end

  def failed? 
    !passed? 
  end
end
@dchelimsky

This comment has been minimized.

Show comment
Hide comment
@dchelimsky

dchelimsky Jun 5, 2011

Member

Again, the example isn't really finished until after all the after hooks have run, so trying to hook this in an after block is unreliable, even if it's a method on the example or example_group.

After a bit more thought, one mechanism that is supported right now is custom formatters. You can use an arbitrary number of formatters, so you are free to do something like this:

class FailureReporter
  def example_failed(example)
    p "#{example.full_description} failed: #{example.execution_result[:exception].message}"
  end

  def method_missing(m, *a, &b)
    # ignore other messages
  end
end

RSpec.configure do |c|
  c.add_formatter FailureReporter
end

describe "something" do
  it "does something" do
    raise "oops, something went wrong"
  end
end
$ rspec example_spec.rb -fd

something
  does something (FAILED - 1)
"something does something failed: oops, something went wrong"

Does that satisfy your need?

Member

dchelimsky commented Jun 5, 2011

Again, the example isn't really finished until after all the after hooks have run, so trying to hook this in an after block is unreliable, even if it's a method on the example or example_group.

After a bit more thought, one mechanism that is supported right now is custom formatters. You can use an arbitrary number of formatters, so you are free to do something like this:

class FailureReporter
  def example_failed(example)
    p "#{example.full_description} failed: #{example.execution_result[:exception].message}"
  end

  def method_missing(m, *a, &b)
    # ignore other messages
  end
end

RSpec.configure do |c|
  c.add_formatter FailureReporter
end

describe "something" do
  it "does something" do
    raise "oops, something went wrong"
  end
end
$ rspec example_spec.rb -fd

something
  does something (FAILED - 1)
"something does something failed: oops, something went wrong"

Does that satisfy your need?

@natritmeyer

This comment has been minimized.

Show comment
Hide comment
@natritmeyer

natritmeyer Jun 6, 2011

An interesting idea, but a formatter will apply to all the scenarios being executed, not just one describe block. Also, I'm not sure if instance variables from scenarios are available to formatters (what I really want to do is to write out an instance variable on scenario failure in only one spec.rb file).

So an example isn't finished until the after block is finished. But... if an exception occured in the body of the scenario then the the scenario has failed and nothing that occurs in the after block will change that. So though it may not be possible to know if a scenario has passed, it is possible to know that it has failed, right? If so, it would not be unreasonable to have a #failed? method available in the after block (whether on the example or example group)...right?

natritmeyer commented Jun 6, 2011

An interesting idea, but a formatter will apply to all the scenarios being executed, not just one describe block. Also, I'm not sure if instance variables from scenarios are available to formatters (what I really want to do is to write out an instance variable on scenario failure in only one spec.rb file).

So an example isn't finished until the after block is finished. But... if an exception occured in the body of the scenario then the the scenario has failed and nothing that occurs in the after block will change that. So though it may not be possible to know if a scenario has passed, it is possible to know that it has failed, right? If so, it would not be unreasonable to have a #failed? method available in the after block (whether on the example or example group)...right?

@dchelimsky

This comment has been minimized.

Show comment
Hide comment
@dchelimsky

dchelimsky Jun 6, 2011

Member

Also, I'm not sure if instance variables from scenarios are available to formatters

Instance variables are not available to formatters.

So though it may not be possible to know if a scenario has passed, it is possible to know that it has failed, right?

But then you'd get a false negative if the failure occurs in a subsequent after block. That might be OK for your situation, but it's not OK for a general purpose addition to the core API.

What if we add an accessor for the @exception instance variable? This wouldn't give you the nice pass/fail method names, but at least you'd be able to check to see if there was an exception more gracefully:

after(:each) do
  if example.exception
    # do something
  end
end

WDYT?

Member

dchelimsky commented Jun 6, 2011

Also, I'm not sure if instance variables from scenarios are available to formatters

Instance variables are not available to formatters.

So though it may not be possible to know if a scenario has passed, it is possible to know that it has failed, right?

But then you'd get a false negative if the failure occurs in a subsequent after block. That might be OK for your situation, but it's not OK for a general purpose addition to the core API.

What if we add an accessor for the @exception instance variable? This wouldn't give you the nice pass/fail method names, but at least you'd be able to check to see if there was an exception more gracefully:

after(:each) do
  if example.exception
    # do something
  end
end

WDYT?

@natritmeyer

This comment has been minimized.

Show comment
Hide comment
@natritmeyer

natritmeyer Jun 6, 2011

Interesting. Thinking about the false-fail aspect of it, I guess the failed? method is conceptually closer to failed_yet?.

Exposing the exception would work. You're right - it's not as pretty as failed?, but it would do the trick.

natritmeyer commented Jun 6, 2011

Interesting. Thinking about the false-fail aspect of it, I guess the failed? method is conceptually closer to failed_yet?.

Exposing the exception would work. You're right - it's not as pretty as failed?, but it would do the trick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment