TestRule impls are not notified of test failure

[ldaley]

Can be verified by using a JUnit TestWatcher impl that collects errors. No errors will be collected, because the Statement given to the rule will never throw.

[leonard84]

There is a known issue for TestRule https://code.google.com/p/spock/issues/detail?id=240

When a TestRule swallows an exception thrown from a test method, the test will nevertheless be reported as failed by Spock. We should change this so that it behaves like in JUnit.

Another difference is that with Spock, base.evaluate() will not throw an exception even if the test method did. This means that TestWatcher rules won't work correctly.

[johnroach]

Hi,
I have ran across this issue. Is there a fix for it in release "1.1-groovy-2.4-rc-1" ? Or any releases at all? This is really hurting us from using a lot of reporting tools.

[johnroach]

I am willing to work on this if someone can point me to the problem... @alkemist @leonard84

[johnroach]

Seriously, free help here! @alkemist @leonard84 just need you guys to show me where to start looking...

[leonard84]

@johnroach thanks for your offer, but I can only point you at the already established facts in my comment above. The main problem is that the exceptions are swallowed and logged before they reach the TestRule. The code which delegates to the TestRules is in org.spockframework.runtime.extension.builtin.TestRuleInterceptor and the initial invocation Object is created here org.spockframework.runtime.BaseSpecRunner#invoke.

Good Luck

[goncalosoares]

Is this planned to be fixed for version 1.1? :-)

[mpetko]

Looks like this issue was not fixed in version 1.1. Is there a plan for fixing this?

[leonard84]

The way TestRule and MethodRule expect exceptions is incompatible with Spocks extension model which is similar to JUnit 5 Jupiters extensions. You'll probably need to write a Spock extension if you need access to the exception. If you are using Spring then you could also use a TestExecutionListener. However, if you want to swallow the Exception then a Spock extension is the way to go.

I'm closing this as wont fix.

[goncalosoares]

Just a quick question as I am pretty new to this :-)
I should write a spock extension to what? Any specific class?

Sorry the beginner question :-)

[leonard84]

http://spockframework.org/spock/docs/1.1/all_in_one.html#_writing_custom_extensions