New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TestRule impls are not notified of test failure #118

Closed
ldaley opened this Issue Aug 18, 2015 · 10 comments

Comments

Projects
None yet
5 participants
@ldaley
Member

ldaley commented Aug 18, 2015

Can be verified by using a JUnit TestWatcher impl that collects errors. No errors will be collected, because the Statement given to the rule will never throw.

@leonard84

This comment has been minimized.

Member

leonard84 commented Aug 18, 2015

There is a known issue for TestRule https://code.google.com/p/spock/issues/detail?id=240

When a TestRule swallows an exception thrown from a test method, the test will nevertheless be reported as failed by Spock. We should change this so that it behaves like in JUnit.

Another difference is that with Spock, base.evaluate() will not throw an exception even if the test method did. This means that TestWatcher rules won't work correctly.

@johnroach

This comment has been minimized.

johnroach commented Jul 11, 2016

Hi,
I have ran across this issue. Is there a fix for it in release "1.1-groovy-2.4-rc-1" ? Or any releases at all? This is really hurting us from using a lot of reporting tools.

@johnroach

This comment has been minimized.

johnroach commented Jul 26, 2016

I am willing to work on this if someone can point me to the problem... @alkemist @leonard84

@johnroach

This comment has been minimized.

johnroach commented Aug 8, 2016

Seriously, free help here! @alkemist @leonard84 just need you guys to show me where to start looking...

@leonard84

This comment has been minimized.

Member

leonard84 commented Aug 8, 2016

@johnroach thanks for your offer, but I can only point you at the already established facts in my comment above. The main problem is that the exceptions are swallowed and logged before they reach the TestRule. The code which delegates to the TestRules is in org.spockframework.runtime.extension.builtin.TestRuleInterceptor and the initial invocation Object is created here org.spockframework.runtime.BaseSpecRunner#invoke.

Good Luck

@goncalosoares

This comment has been minimized.

goncalosoares commented Apr 27, 2017

Is this planned to be fixed for version 1.1? :-)

@mpetko

This comment has been minimized.

mpetko commented May 31, 2017

Looks like this issue was not fixed in version 1.1. Is there a plan for fixing this?

@leonard84

This comment has been minimized.

Member

leonard84 commented May 31, 2017

The way TestRule and MethodRule expect exceptions is incompatible with Spocks extension model which is similar to JUnit 5 Jupiters extensions. You'll probably need to write a Spock extension if you need access to the exception. If you are using Spring then you could also use a TestExecutionListener. However, if you want to swallow the Exception then a Spock extension is the way to go.

I'm closing this as wont fix.

@leonard84 leonard84 closed this May 31, 2017

@leonard84 leonard84 added the won't fix label May 31, 2017

@goncalosoares

This comment has been minimized.

goncalosoares commented Jun 14, 2017

Just a quick question as I am pretty new to this :-)
I should write a spock extension to what? Any specific class?

Sorry the beginner question :-)

jjohannes added a commit to gradle/gradle that referenced this issue Jul 24, 2017

Switch test directory provider to from class to method rule
When interacting with Spock, class rules might not be aware of failing
tests, because the fail is not throwing.
(see spockframework/spock#118)
For method rules however, this is working in Spock 1.0.

This fixes the issue that the rule was deleting information important
for failure analysis (e.g. logs of isolated daemons) with the test
directory.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment