Historically, android-junit5 has been skipping test execution when instrumentation tests are executed on an Android device that doesn't meet the minSdk requirements of JUnit 5. This might cause issues to go unnoticed and that could be surprising – ref this recent problem in OkHttp.
We could consider changing the default behavior so that execution fails on devices < API 26, instead of failing silently. At the very least, this could be a customization that can be configured.