bug in ignore an overriding method

[inbarsto]

i dont know if that is a bug or a "feature" but when tring to @OverRide a method and then putting that overridden method in @ignore then the method is not Ignored unless the parent method is also Ignored...

[marcphilipp]

Can you provide an example?

[inbarsto]

say you have this strocture:

public abstract Foo extends otherFoo{

  @Test
  public void doSomeThing(){
  }
  .
  .
  .

}

public class sonOfFoo extends Foo{

@Ignore
@Test
public void doSomeThing(){
.
.
}

}

when running a junit test on the entire class sunOfFoo then doSomeThing is not ignored....

[inbarsto]

could that be by design that the method under Ignore is "overlooked" in run time?

[marcphilipp]

How do you run your test, what version of JUnit are you using? I
cannot reproduce the issue using Eclipse + JUnit 4.8, where the method
is ignored, as expected.

[inbarsto]

i run on intellij 11.1.3 jdk1.6.0_31 junit 4.8(i think)

[Tibor17]

you have to test it by using JUnitCore or maven , because idea is using own runners and there could be the problem.

[dweiss]

JUnit does have a flaw in proper (?) detection of method inheritance. For example running Sub:

public static class Super {
  @Test
  public void testMethod() {}
}

public static class Sub extends Super {
  @Ignore
  public void testMethod() {
    System.out.println("Hello");
  }
}

should be ignored and isn't (because @test method is retrieved from the superclass and its invocation delegates to the subclass which isn't considered by JUnit. Similar thing happens when you open up a @test method's scope to public (from protected or package private) in a subclass. This is by-design I guess since no virtual method analysis is done.

[marcphilipp]

@dsaff Can you shed some light upon this issue?

[dweiss]

JUnit considers the first bottom-up @test method as the dispatch point and source of other annotations. This isn't necessarily true as demonstrated by the above snippet of code -- to me, the bottom of the actual virtual dispatch should be considered, not which particular "parent" overridden method is annotated. Examples could be multiplied, for instance a parent abstract class specifying

@test protected abstract void testMethod() {}

and a concrete subclass opening up this method:

public void testMethod() {}

(not missing Test annotation which should be "inherited" from the overridden method). This will fail saying @test method must be public (and it is in a conrete subclass).

A contract where method annotations are collected from actual virtual dispatch chain seems clearer to me; this also includes method shadowing. Otherwise you get inconsistencies like this one:

public class Super {
@BeforeClass public static void beforeClass1() { sysout("super.beforeclass1"); }
public static void beforeClass2() { sysout("super.beforeclass2"); }
@BeforeClass public static void beforeClass3() { sysout("super.beforeclass3"); }

@Test        public void testMethod() {} 

}

public class Sub extends Super {
public static void beforeClass1() { sysout("sub.beforeclass1"); }
@BeforeClass public static void beforeClass2() { sysout("sub.beforeclass2"); }
@BeforeClass public static void beforeClass3() { sysout("sub.beforeclass3"); }
}

Which static hooks will be invoked by JUnit? All of them are shadowed in Sub yet beforeclass1 will be called on super and beforeclass3 only on sub (because beforeclass1-shadowed method doesn't have @BeforeClass annotation).

This is inconsistent to me. It should be that annotations are pseudo-inheritable that is apply to shadowed or overridden methods.

[Tibor17]

@inbarsto
I used your examples and u are using junit wrong.
The test method (on top sub class) is ignored if both annotations appear on it:
@test @ignore
public void myTest() {}

Having only @ignore is not anough. This makes sense in real situations where you write a test and annotate it by @test. After implementation became buggy you ignored the test by @ignore annotation.
This happens in real when you want to make the build passed. I do not say it is a good to ignore tests instead of fixing implementation of main source code, but this is more technical.

In junit BlockJUnit4ClassRunner is your runner, and #runChild operates on test methods annotated by @test. If the test method is ignored at the same time, then ignored tests counter is incremented.

The TestClass in junit populates all methods from sub to super and avoids populating overridden methods in sub classes if they already appeared in super class with same method signature.

[dsaff]

I would not be surprised if the way in which JUnit currently handles this is inconsistent or surprising--we've had repeated issues with trying to layer an annotation-based model on top of a language with overridable methods.

I'd like to tackle the issues one at a time--there seems to be some inconsistencies in how different users are seeing the code being implemented. @inbarsto, can you paste an actual compilable snippet that reveals your bug?

[Tibor17]

If i use @ignore by javadoc, all is fine. But the example from @dweiss is a problem.
According to our code, it seems that this problem should appear with annotations Theory, Rule, etc. as well.
In the code, the TestClass is analyzing class inheritance, and doing mapping annotation -> FrameworkMethod.
If the mapping was opposite, then the inheritance would work, right?

[dweiss]

I spent some time on this when I was implementing a runner for randomized testing package. My conclusion was that to do it "right" you need to analyze the scoping, inheritance and shadowing rules much like the JVM spec tells you to do and pick the "bottom" method of a shadowing/ virtual dispatch chain, then look for any annotations going up. Probably the "cleanest" solution for dispatch would then be to run the leaf methods in such chains but this will break current JUnit behavior for static methods (see my example above -- beforeClass1).

I've implemented a method hierarchy model recently that tries to mimic JUnit behavior. It isn't perfect but it allows walking those "chains" of shadow and inheritance relationships between methods. The code is fairly new and raw but if you want to peek, feel free to.

https://github.com/carrotsearch/randomizedtesting/blob/master/randomized-runner/src/main/java/com/carrotsearch/randomizedtesting/ClassModel.java

The getAnnotatedLeafMethods is what you're after. It returns the closest-to-the-bottom annotated method in each chain. Again -- ideally we would return just the bottom method but it's not what JUnit does now and for reasons of backwards compatibility I made it identical.