Ignore by default all tests with org.junit.Ignore annotation

[CristianGM]

Tests annotated with ignore will cause an infinite loop if not ignored, to prevent that we could ignore them by default on the Gradle Plugin.

[tagantroy]

@CristianGM Can you reproduce it on sample project?

Device pool omni: 24 passed, 0 failed, 2 ignored tests

on latest develop branch it works as expected

[lukaville]

We also have the same problem, it doesn't hang with blacklisted org.junit.Ignore annotation

[lukaville]

@tagantroy
Looks like it's happening when you place @Ignore annotation on the class, not the test method.

[CristianGM]

I had the same issue with the annotation on the method and the class.
I thought the reason was that the method had more than one annotation and maybe the plugin was looking just for the first annotation.

[tagantroy]

https://github.com/Malinskiy/marathon/tree/reproduce_problem_with_Ignore_on_class.
I reproduced this problem and found interesting logs

java.lang.NullPointerException: Attempt to invoke interface method 'int java.lang.CharSequence.length()' on a null object reference
at java.util.regex.Matcher.reset(Matcher.java:995)
at java.util.regex.Matcher.<init>(Matcher.java:174)
at java.util.regex.Pattern.matcher(Pattern.java:1006)
at android.support.test.internal.runner.TestRequestBuilder$MethodFilter.stripParameterizedSuffix(TestRequestBuilder.java:524)
at android.support.test.internal.runner.TestRequestBuilder$MethodFilter.evaluateTest(TestRequestBuilder.java:510)
at android.support.test.internal.runner.TestRequestBuilder$ParentFilter.shouldRun(TestRequestBuilder.java:145)
at android.support.test.internal.runner.TestRequestBuilder$ClassAndMethodFilter.evaluateTest(TestRequestBuilder.java:452)
at android.support.test.internal.runner.TestRequestBuilder$ParentFilter.shouldRun(TestRequestBuilder.java:145)
at org.junit.runner.manipulation.Filter$3.shouldRun(Filter.java:112)
at org.junit.runners.ParentRunner.shouldRun(ParentRunner.java:434)
at org.junit.runners.ParentRunner.filter(ParentRunner.java:382)
at org.junit.runner.manipulation.Filter.apply(Filter.java:97)
at android.support.test.internal.runner.TestRequestBuilder$LenientFilterRequest.getRunner(TestRequestBuilder.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:58)
at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:375)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1932)

I also found this issue jaredsburrows/gradle-spoon-plugin#47

[Malinskiy]

We're going to check the Ignored tests after parsing all the tests from vendor module during the init phase and we're going the create a fake report for these, because real test runner doesn't behave as expected.

[Malinskiy]

I've checked the implementation of the Android test runner and found other ways to ignore the test but using the @Suppress for example will not report the test at all

The problematic bit in the runner is checking if something is a test or not. The logic is that it's a test if there are no children in object tree and that usually happens, but in this case this condition is also true for the whole class and then the parameterized logic crashes because it assumes that it's input is CLASS#METHOD but in reality it's just CLASS and hence NPE for the method part

[Malinskiy]

The reason why it is considered a test is because the runner implementation that is used in case of ignored class is IgnoredClassRunner compared to the usual Suite. The difference is that Suite actually returns children for the class node hence the isTest passes properly

IgnoredClassRunner
Suite