Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Feature: Add ability to bail from current suite only #2224
I have a lot of tests that are dependent on previous steps, and when an early step fails it causes everything after it to fail too, leading to a lot of unnecessary noise. There's only so much that makes sense to go in the before() block. I've implemented the feature myself, but the contributions guidelines say not to PR new features, so I'll just link to my fork here: master...hoverduck:master
It's a relatively simple change, but it's possible (likely) I missed something, so I'd love feedback even if you opt not to accept a PR for it.
My implementation works by calling
Interesting idea! I realize there's def times where it's hard to manage all the dependencies and requirements for a spec. It's generally considered an anti-pattern to have tests depend on the execution of other tests - a good test suite should be able to have its individual specs run in any order. But as you've highlighted, you can end up with a lot of noise. I generally see this as being a more common issue with acceptance tests using phanntomjs/selenium/nightmare/etc.
That said, I'm not sure I really see the benefit of bailSuite. If you want reduced noise, shouldn't you just use bail? Why continue to run the test suite at all, since bailSuite ends up masking the total number of errors in your suite. Just because the first error'ed doesn't mean others in the test suite would have. So the reporting can't be accurate?
I'm thinking this is something that would make for a good plugin though! :)
Exactly right. We're running end-to-end acceptance tests of our system.
Bail reduces the noise too much. I'd rather run all the later tests to know whether I have more problems. Given the choice between too much noise and not running all the test suites, I'd rather run all the suites.
I prefer to think of it as squashing all of the errors into a single report. You could make the argument that this is basically duplicating the functionality of the before() hook, but I think this just gives some extra flexibility.
Sounds good to me. Is this (#1457) the current status of the plugin system? What's the release schedule look like for that?