@Rule allow to ignore Tests

[RainerW]

Is there a way to create the same behaviour then @ignore with a rule?

This would allow me to create a custom annoationen and rule where you can mark tests which cannot be executed in a given Environment.

[digulla]

I have a similar problem. In my case, I have fast and slow tests. Slow tests build a database, fill it with data, etc. Those take several seconds or even minutes to run.

Right now, I'm using a custom runner to enable/disable them with a System property but a rule would be much better.

So please add a property to FrameworkMethod called "ignore". When the instance is created, set this to true if an @ignore annotation is present.

Add the usual getters and setters.

In BlockJUnit4ClassRunner and similar places, replace "method.getAnnotation(Ignore.class)" with "method.isIgnore()".

This would allow MethodRule and other JUnit extensions to flag a test method as ignored.

As it is, I can ignore a test method in a MethodRule by returning an empty Statement but in the various JUnit runners, it now looks as if the method was executed successfully which isn't correct.

[dsaff]

We intend for tests that throw AssumptionFailedException to be marked as ignored by runners. Does that work for your case?

[digulla]

At least, it's a workaround. I can move my checks into a helper method so the tests only needs a single line of code (DRY principle).

I'd prefer an annotation, though, because that would allow to easily create a report from the source. Can I throw that exception from a Statement.evaluate(), too?

[dsaff]

Yes, you can, so your assumptions can be bound up into a Rule.

[RainerW]

I use the Assumption workaround for some time now. The main drawback is that the decision if the Test is ignored, happens when already code is getting executed.

E.g. I have a selenium server rule, which starts a selenium server. Which is not the fastes operation. when now my rule detects i'm running in a non local Environment it will skip all upload tests. Sadly even in this cases the server will start.

So having statement.isIgnored() called before the actual execution, would same plenty of time.

[digulla]

Am 24.10.2011 20:05, schrieb RainerW:

I use the Assumption workaround for some time now. The main drawback is that the decision if the Test is ignored, happens when already code is getting executed.

E.g. I have a selenium server rule, which starts a selenium server. Which is not the fastes operation. when now my rule detects i'm running in a non local Environment it will skip all upload tests. Sadly even in this cases the server will start.

So having statement.isIgnored() called before the actual execution, would same plenty of time.

Try to throw the assumption in @before or even @BeforeClass, i.e. in the
code that starts the server. Should work.

Regards,

Aaron "Optimizer" Digulla a.k.a. Philmann Dark
"It's not the universe that's limited, it's our imagination.
Follow me and I'll show you something beyond the limits."
http://blog.pdark.de/

[RainerW]

the execution order is:

• beforeClass ( here i cannot decide on a testmethod basis, only on a testclass level )
• rules[] -> before() ( this is where selenium starts )
• before ( this is where i detect the wrong environment )
• test + afters

So throwing the Assumtion in @before doesn't do the trick, it's already too late. But the environment detection is a rule anyway. So i could define the rules in a given order to circumvent this.But still i think it would be nice to skip tests with a method and not with an Exception.

[dsaff]

RainerW, have you tried the Categories support? Will that do what you want?

[RainerW]

I tried Categories, this implies that the test Executor (human Buildsystem) allready knows what can be executed. in the Selenium Sample i would like to have the logic via certain Annotations at the test. When they evaluat to X the test get's ignored.

A extreme form of this is ignoring Tests with open tickets, only Tests with a @IssueID("fixed:JIRA-123") are getting executed. This would lead to a massive set of categories.

Btw. because most "Viewers" show an Assumtion as "Green" this the assumtion solutions is only a bad workaround. This is why i now use my own runner for better feedback.

( Personally i do not use Categories; because: a) it needs a runner oder b) need to pass a start parameter and c) i hate to have interfaces just lying around )

[digulla]

Works for me. You can close this.

[Tibor17]

@RainerW , @digulla
I am convinced you will not need any extra Rule to create.
In the issue #142 you can ignore/exclude tests

• by creating own Filter and run your suite TestSuite and control it by build process as first alternative
  CategoryFilter filter= CategoryFilter.categoryFilter(null, ANY, Collections.singleton(B.class, C.class), ANY)
  Request request= Request.aClass(TestSuite.class);
  Result result= new JUnitCore().run(request.filterWith(filter));
• or as the second alternative would be to use system properties
  -Dorg.junit.categories.excluded=/B.java,/C.java

So your suite would be as follows:

@RunWith(Categories.class)
//NULL represents categories without limitation to included.
@IncludeCategory(value = null, assignableTo = Selection.ANY)
@SuiteClasses({A.class, B.class, C.class})
public class TestSuite {}

It's enough to annotate every test class by itself, e.g.:

@category(A.class)
public class A
{...@test...}

[RainerW]

I think this discussen got a litte bit of track.

My original question was how could a Rule create the same behaviour as @ignore.

The suggestet Solutions so far:

a) throw AssumtionError in your rule

drawback 1 :
i see a difference between Ignore and a Assumtion.
-> Ignore, means ignore, Assumtion means not today/or not on this operation-system

drawback 2 :
Rules just use a decorator pattern. Even when i throw a AssumtionError in my rule's code, i cannot be sure that NO other rule had been execute before me and therefore allready done their before-test code.

b)throw AassumtionError in @before

drawback :
this makes the delegate concept of Rules useless, so this is not a solution.

c) use Categories

drawback 1:
This moves the logic away form my rule into categories. ( I share Rules between projects, but not Categories! )

drawback 2:
I need to use the Categories Runner ( Not compatible with 3rd party runners, eg. SpringTestRunner or some other runners i need for performance measurements)

idea 1

A possible solution would be an additional Method in the MethodRule/TestRule interface:

/** @return true if the TestMethod is actually ignored */
boolean isTestIgnored( FrameworkMethod method, Object target );

This method should then be called by the Runner before any apply method get's executet. Meaning there would be a kind of additional "Test"-Phase :

setupTestMethods() // -> query all rules for ignore methods etc.
before() // same as now, execute @before code of rules & test-classes
execute() // execute Test
after() // same as now, execute @after code of rules & test-classes

idea 2

Create an additional concept, similar to Rules, which can "Extend" Runners. This Extension can then inject logic to ignore TestMethods. And I guess adding an additional Test-Call could also be a nice feature which would be fit in such 'extension-point'.

idea 1+2

Move the isTestIgnored() Method from idea #1 into a interface. Runners now can sill query all Rules in the test-setup-phase for additional ignore-testmethods, but only Rules which actually use this would implement this interface.

[digulla]

Thanks for the great summary, RainerW. Just one point though: Making sure that the ignore rules get executed first might not be a good idea - the ignore rule might need information from other rules.

[RainerW]

This should only happen if you have closed coupled rules, which i cannot think of a scenary at all.

But execution of MethodRule::apply() and Statment::evaluate() are separated, so if this is really a problem, you need to initialize that part inside MethodRule::apply().