Android Library seemingly can't filter Instrumented tests via @Tag

[ktkopone]

I'm trying to prototype a simple Hello World gradle android library project to get the @Tag filtering working with some tests. After some amount of hunting around, I got this working with unit tests (though as an aside, https://github.com/mannodermaus/android-junit5/wiki/Migrating-from-1.0.x is a bit obscure and should probably be pulled into the main wiki pages, e.g. android.testOptions.junitPlatform.filters.tags.include having been moved to <...>.filters.includeTags was a pain point). Thus running e.g. ./gradlew clean test is now successfully only running tests tagged with @Tag('sampleTag').

However, instrumentation tests are still all being run regardless of the filter, e.g. ./gradlew clean connectedAndroidTest runs both tests regardless of the @Tag('sampleTag')

The following is my build.grade:

build.gradle.txt

and my project is laid out with unit tests in src/test/java and instrumentation tests in src/androidTest/java. This is the current instrumentation test file I'm running:

StupidConnectedTest.java.txt

Is this not currently supported, or am I missing some separate place where filters on instrumentation tests need to be defined, or a gradle dependency, etc?

Please and thanks for any help!

EDIT: Sorry Tom g, didn't mean to tag you.

[ktkopone]

Also a minor side question in case you happen to know: is there any way to reduce verbosity of instrumentation test output? Atm the output for just 2 passing tests is this monstrosity:

02:52:27 V/ddms: execute: running am instrument -w -r   -e runnerBuilder de.mannodermaus.junit5.AndroidJUnit5Builder com.stupidSampleGradleProject.AppName.test/androidx.test.runner.AndroidJUnitRunner
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: class=StupidConnectedTest
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: current=1
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: numtests=2
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: stream=
02:52:30 V/InstrumentationResultParser: StupidConnectedTest:
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: test=stupidConnectedFailTest()
Starting 2 tests on Nexus 5X - 8.1.0
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS_CODE: 1
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: class=StupidConnectedTest
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: current=1
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: numtests=2
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: stream=.
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: test=stupidConnectedFailTest()
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS_CODE: 0
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: class=StupidConnectedTest
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: current=2
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: numtests=2
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: stream=
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: test=stupidConnectedPassTest()
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS_CODE: 1
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: class=StupidConnectedTest
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: current=2
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: numtests=2
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: stream=.
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS: test=stupidConnectedPassTest()
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_STATUS_CODE: 0
02:52:30 V/InstrumentationResultParser: INSTRUMENTATION_RESULT: stream=

[mannodermaus]

Thanks for the detailed information!

The backwards compatibility layer behind the instrumentation test support for JUnit 5 disregards @Tag annotations - they are currently only honored by unit tests, as you have already pointed out. I've had a POC in the works to enable support for includeTags/excludeTags on instrumentation tests, which was revitalized for the first SNAPSHOT of version 1.5.0. It re-implements some of the backwards compatibility layer to not be dependent on the JUnit Team's module, so that we can let the Platform filter out instrumentation tests. However, it will only acknowledge tags for the time being - engines and class name patterns are not used for instrumentation tests.

It'd be awesome if you would try this on the snapshot I just uploaded. You'll want to use the following dependencies:

// build.gradle:

buildscript {
  repositories {
    maven { url "https://oss.sonatype.org/content/repositories/snapshots/" }
  }
  dependencies {
    classpath "de.mannodermaus.gradle.plugins:android-junit5:1.5.0.0-SNAPSHOT"
  }
}

allprojects {
  repositories {
    maven { url "https://oss.sonatype.org/content/repositories/snapshots/" }
  }
}

// app/build.gradle:

dependencies {
  androidTestRuntimeOnly "de.mannodermaus.junit5:android-test-runner:1.1.0-SNAPSHOT"
}

Let me know if this works for you!

[ktkopone]

Seems to be working well, thanks! Only nit is that when a non-existent tag is set, connected tests (but not unit tests) seem to run one 'null' test instead of saying '0 tests run', but I don't know if that's an issue with this or with gradle/android.

Also for posterity of anyone reading this, I set the android-test-runner runtime dependency in the main build.gradle, since our project is an android library instead of an android app and thus has no app directory. Here's what the build.gradle looks like for basic filtering of unit and instrumentation tests, with just a single tag settable via a passed-in property (e.g. ./gradlew clean test connectedAndroidTest -Ptag=sampleTag):

build.gradle.txt

Any idea when a non-snapshot for this version will be released? Also, any idea how to reduce instrumentation test output verbosity as mentioned above? I've poked around but have only figured out how to control unit test output.

Thanks!

[mannodermaus]

connected tests (but not unit tests) seem to run one 'null' test instead of saying '0 tests run'

I've seen this as well, but debugging the events sent to the RunNotifier, there are no superfluous messages sent to the build tool. As far as I know, this should be an IDE issue, unfortunately.

Thanks again for providing the sample info! This should be helpful for others seeking to be at the forefront of this stuff as well.

Any idea when a non-snapshot for this version will be released?

There's a couple of tweaks I'd like to apply to the new support (in terms of DSL) before submitting the final release. No hard promises, but I'm aiming for the end of this week ~ I'll keep you posted!

any idea how to reduce instrumentation test output verbosity as mentioned above?

Unfortunately not from the perspective of preventing these logs from being emitted. You could raise the log level of Logcat or apply a filter to the output, I suppose.

[mannodermaus]

This has been released with Gradle Plugin 1.5.0.0 and android-test-runner 1.1.0. Note that you need to update both coordinates to make use of this, and also I'd like to reiterate that the filtering only works for @Tag-based filters for now. Feel free to open a new issue if you run into any problems!