RSpec fails to return a proper error code when a capybara test fails #410

merged 5 commits into from Aug 25, 2011

4 participants


This is a workaround for jnicklas/capybara#178. Capybara has an at_exit call that messes up the status code returned from the process. This workaround ensures that at_exit from rspec is run last (ie. is setup first).

The same was applied in minitest seattlerb/minitest@979406d.

@dchelimsky dchelimsky closed this Jun 24, 2011

@dchelimsky: you closed this without a reason? this is still a problem that affects a ton of people, no?

RSpec member

Looks like I missed a step - sorry about that. This is a duplicate of #400.


But #400 has a messy fix that was rolled back because it breaks 1.8.7 support. This doesn't as far as I can think of, and works around the problem. No?

@dchelimsky dchelimsky reopened this Jun 30, 2011
RSpec member

I've reopened this to keep it on the radar, but I'm working on a different solution: eliminating the at_exit hook entirely. If that doesn't work out I'll revisit this.

@dchelimsky dchelimsky was assigned Jun 30, 2011
RSpec member

I've been toying with removing the at_exit hook, and it is a bit problem-ridden. I think it's the right way to go in the long run, but I don't want that to hang up the next release.

In the mean time, I merged this and got a failing cuke: @dblock - can you look at getting that scenario to pass?


I was having all kinds of problems doing a repro, so I got a clean rspec-core from git master, running bundle install and rake. Rspec specs pass. Cucumber specs fail with the following.

    process still alive after 5 seconds (ChildProcess::TimeoutError)
    features/command_line/line_number_appended_to_path.feature:97:in `When I run `rspec example_spec.rb:7 --format doc`'

    process still alive after 5 seconds (ChildProcess::TimeoutError)
    features/command_line/line_number_option.feature:55:in `When I run `rspec example_spec.rb --line_number 5 --format doc`'

Many more like this... before I fix real errors, how do I fix this?

RSpec member

The failures you are seeing happen intermittently depending on the environment (they are both timeouts) - although I thought I bumped the time to 15 secs, but maybe not (if not, I should).

As for the failure:

$ bin/cucumber features/command_line/ruby.feature:10 --format pretty
Using the default profile...
Feature: run with ruby command

  You can use the `ruby` command to run specs. You just need to require

  Generally speaking, you're better off using the `rspec` command, which
  requires `rspec/autorun` for you, but some tools only work with the `ruby`

  Scenario:                                                # features/command_line/ruby.feature:10
    Given a file named "example_spec.rb" with:             # aruba-0.4.2/lib/aruba/cucumber.rb:15
      require 'rspec/autorun'

      describe 1 do
        it "is < 2" do
          1.should be < 2
    When I run `ruby example_spec.rb`                      # aruba-0.4.2/lib/aruba/cucumber.rb:52
    Then the output should contain "1 example, 0 failures" # aruba-0.4.2/lib/aruba/cucumber.rb:78
      expected "No examples found.\n\n\nFinished in 0.00003 seconds\n0 examples, 0 failures\n" to include "1 example, 0 failures"
      @@ -1,2 +1,6 @@
      -1 example, 0 failures
      +No examples found.
      +Finished in 0.00003 seconds
      +0 examples, 0 failures
      features/command_line/ruby.feature:22:in `Then the output should contain "1 example, 0 failures"'

Failing Scenarios:
cucumber features/command_line/ruby.feature:10 # Scenario: 

1 scenario (1 failed)
3 steps (1 failed, 2 passed)

Ok. Got a repro. But I don't think we can fix this.

Autorun used to run last by registering itself as an at_exit hook. So now that it doesn't, it no longer runs last. So specs that are describe-d after the inclusion don't exist yet. The only fix is to include autorun last or to go back to the at_exit hook that breaks the original problem.


Also, note to self, I should write a test that simulates the at_exit problem with capybara ...


I just forked rspec-core (from v2.6.4) to merge this patch for our build server, it seems to do the trick:

Any chance of merging dblock's fixes while the proper at_exit fix is in progress?

@dchelimsky dchelimsky added a commit that closed this pull request Aug 22, 2011
@dblock dblock at_exit hax to ensure that rspec's exit code wins.
- Closes #410.
RSpec member

The problem was that the exit code was coming from Capy's at_exit hook, not that rspec's wasn't being run. The patch as/was solved that problem but broke other behavior: running specs with the rcov or ruby commands. I made a minor adjustment in dc290d3 that resolved that issue and now all specs and cukes are passing.

Thanks @dblock.


Doesn't work :( If anything calls at_exit and sets the process exit code (which Capybara does), it overwrites RSpec's. has a cucumber test that reproduces it.
Still don't know how to fix it.

I did some cleanup on my fork, use at-exit-order branch that has the autorun workaround (but breaks autorun).

@dchelimsky dchelimsky added a commit that referenced this pull request Aug 23, 2011
@dchelimsky dchelimsky Restore old at_exit hook. This leaves #410 open and unresolved.
Also tweak the 'upstream at_exit{exit 0}' cuke to fail correctly, but
leave it tagged @wip.
@dchelimsky dchelimsky reopened this Aug 23, 2011
RSpec member

OK - I've reviewed jnicklas/capybara#178 and have a much better understanding of the issue now. I've restored the original at_exit hook, which passes all specs and cukes except for the new one associated with this issue, which I've also restored to something like @dblock's original version which exposes the issue.

I can't accept @dblock's patch as/is, as it breaks other functionality while resolving this issue. I've explored a number of other options and I'm stumped, so if anybody else can come up with a way to resolve this without breaking anything else, you'll make a bunch of people very happy.


@dchelimsky: thank you, totally agree

I'd like to make a radical proposal, - to kill autorun. IMO, it's evil since it runs something automagically. I wonder who or why would use it? So I can't write ruby whatever_test.rb, but I can still do rspec whatever_test.rb which is how most people I know use rspec.

RSpec member

@dblock - not at all radical. That's what I was referring to in my 2nd comment in this thread. The trick is that there are a number of use cases that we need to support. For example, we can't have rspec running automatically just because it is required (it should only run when specs are loaded), and, as you suggest, it needs to run whether you type ruby some_spec.rb or rspec some_spec.rb. I've been playing around with this for a while and have yet to come up with a solution that solves for all the use cases. If you have some ideas about how to do it, I'm all ears/eyes.

RSpec member

@dblock - also - I'm pretty sure that this particular issue (an upstream at_exit hook that exits) will remain if we eliminate our own at_exit hook.


@dchelimsky: half of the problems are because of the at_exit nesting (see #4400). So I would keep at_exit, just not nested. Now, why should spec autorun when I do ruby some_spec.rb?

RSpec member

@dblock - how else will the spec run?


@dchelimsky: rspec some_spec.rb. That uses CommandLine to find the specs and then runs them. So you don't need autorun to do it on an at_exit. In contrast, when you do ruby spec.rb, it includes autorun first (you have to), then registers the spec and runs it on an at_exit. I'll try to find some time to R.I.P autorun out tomorrow - it will be clear what works and what is now gone.

RSpec member

@dblock - I look forward to seeing what you come up with. Keep in mind that we need to be able to run the same file with all of these:

rspec example_spec.rb
ruby example_spec.rb
rcov example_spec.rb

There are also other cases like rspec spec, which will run all the files in the spec directory that match a pattern. And a new feature that let's you type just rspec, which runs all the files in the default_path (defaults to 'spec', but is configurable) that match a pattern.

The other trick is that the specs should not be run by simply requiring 'rspec/core', as that would mean specs run when you run any rake task.

The code you'll need to work with is in lib/rspec/core/runner.rb and lib/rspec/core/configuration.rb.

Good luck!

RSpec member

@dblock - oh, and then there's the rake tasks :) See lib/rspec/core/rake_task.rb.


Yesterday I had the same issue and I fixed this at capybara side jnicklas/capybara#463 but then I found out this pull request here. Maybe this should be fixed at both sides?


@ebeigarts: if your fix actually fixes the problem (looks like it does), then it should be merged to Capybara. We've been going back-and-forth here on trying to fix all scenarios at no avail so far.

@dchelimsky dchelimsky merged commit dc290d3 into rspec:master Aug 25, 2011
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment