Exceptions in Context should fail all tests in scope

[staxmanade]

If I have some code in the Context for setup purposes that fails, it would be great if the project would consider one of the two strategies below:

1. Consider failing each It assertion/test with the same exception.
2. Only report one error (relaying the fact that an error in the context setup failed)

Thoughts?

Example:

$here = Split-Path -Parent $MyInvocation.MyCommand.Path
$sut = (Split-Path -Leaf $MyInvocation.MyCommand.Path).Replace(".Tests.", ".")
. "$here\$sut"

Describe "SomeDescribe" {
  Context "When some Context runs and an error happens" {

      throw "some error during setup"

      It "Should report something" {
          # This test should not even run (but possibly fail with the error from the context setup)
      }
      It "Should report something else" {
          # This test should not even run (but possibly fail with the error from the context setup)
      }
    }
}

[scottmuc]

Interesting scenario, thanks for creating an issue on this. Will look at it in the near future.

[ghostsquad]

I agree with this.

[dlwyatt]

In terms of implementation, it would be much easier to output one message and then ignore the rest of the current Context (or Describe, if the error happens within that scope), than it would be to try to output messages for each subsequent skipped test.

What would be the value of having all of the tests that weren't executed in the output? If it's worth the development effort, then it could certainly be done. Also, what would be the proper behavior if an error occurs in a Describe which contains one or more Context statements? Should the aborted Context statements and the It statements they contain be output as well?

[mzs-github]

For CI jobs it would be handy if Pester could exit with a non-zero exit code if an exception is thrown from a Context (assuming it was invoked with -EnableExit). Currently it exits with code 0 as long as all tests that could run have passed.

Counting the exact number of skipped tests is probably less important than reporting that there were errors. Just for reference I looked into what JUnit does when a @BeforeClass method fails - it reports a single error and doesn't bother counting the individual @Tests it missed.

[dlwyatt]

That should be pretty easy to do. I'll look into it this week.

[dlwyatt]

Just opened a very basic PR to accomplish this, to give us something concrete to talk about. There may be improvements to be made here. (For instance, the code that parses BeforeEach / AfterEach blocks inside a Describe would still terminate the whole script in the "old way" if there's a syntax error; that could be moved into the Try block to produce a failed test instead.)

[dlwyatt]

Merged the PR which turns a terminating error in a Describe or Context into a single failed test (in addition to whatever else already happened in that block prior to the error.) This will be part of the upcoming v3.0 release.