Problem finding tests in IntelliJ using Gradle test runner

[sebersole]

No idea if this is a IntelliJ, Gradle or JUnit issue. I've tried with both JUnit 5.4 and 5.7 - same issue. Ditto for JUnit 4 btw

For a reproducer, this is happening with the Hibernate project (http://github.com/hibernate/hibernate-orm) with the wip/6.0 branch. Totally understand if you want a smaller reproducer - let me know.

I played around a little bit to see if I could narrow down possible causes. It seems like the projects that have multiple test tasks (how we expose testing the different available databases) are the ones experiencing this problem. That is most of the Java sub-projects.

To reproduce:

1. Set IntelliJ to run tests using Gradle (Settings > Build, Execution, Deployment > Build tools > Gradle : "Run tests using : Gradle"). I see the same behavior with "Chose per test"
2. Select an individual test class/method from a Java sub-project (e.g. hibernate-core) and run or debug it

I expect the tests to run. IJ fails to find them. E.g.

Execution failed for task ':documentation:matrix_derby'.
> No tests found for given includes: [org.hibernate.userguide.mapping.basic.NationalizedStringMappingTests](filter.includeTestsMatching)

As a counter-point, if you use that same IJ set-up and try to do the same with tests in the hibernate-testing sub-project, it works fine. That project has just the one test task.

[marcphilipp]

I checked out the project and tried running a few tests. I think there are different problems here:

:documentation:matrix_derby is using JUnit 4 not JUnit Platform, but probably not on purpose? I think the each should be a configureEach in java-module.gradle.

When I tried running :documentation:test with StringNationalizedMappingTests I got a failure for the :hibernate-testing:test task. I think that's an IntelliJ issue that passes the class name as a filter to the wrong/all test tasks. Usually, that's not a problem, because test tasks don't depend on other test tasks but in your case they do. I think you should open an IntelliJ issue for that one.

In any case, I'm convinced this is not an issue with JUnit Platform, Jupiter, or Vintage and hence I'm closing this issue.

[sebersole]

Well this is disappointing...

Yep, as I said... it happens regardless of whether the tests use JUnit 4 or 5.

Regarding the file you link, I'm not sure why you think that links the test tasks in question together. hibernate-testing is a bunch of test-fixtures. It defines some simple tests for its fixtures (basically our JUnit 5 extensions). That section you linked is basically saying that for every java project (other than hibernate-testing itself), have its test tasks depend on running the test-fixture tests. Not idea why you think that is "unusual".

Regarding each / forEach... not a Groovy expert so 🤷 I'll research a bit. Again though, why would that have this (not being able to run tests) impact?

[sebersole]

I guess we will have to re-think using JUnit 5 then. Not being able to run and debug tests against the various databases is just a killer, regardless of where the fault lies

[marcphilipp]

This is not an issue with using JUnit 5. The same would happen if you would only use JUnit 4 and run tests from IDEA via Gradle.

[marcphilipp]

For test fixtures, you should consider using Gradle's java-test-fixtures plugin.

[marcphilipp]

I created a reproducer raised an issue with IDEA: https://youtrack.jetbrains.com/issue/IDEA-269368

[sebersole]

We discussed using the test-fixtures plugin. But as we understand it, it simply adds that as a variant, and the only way to map that to POM is via classifier. Classifier simply cannot model this; you'd have to expose your test dependencies as "main dependencies". Considering we need to support people consuming Hibernate via Maven, that will not work. Unless I have missed something.

Thanks for opening the issue. I tried removing the dependency to hibernate-testing:test. I can now run test for the various sub-projects, but not the database specific tests (matix_derby, etc.). I'll do some more investigating

[sebersole]

This is not an issue with using JUnit 5. The same would happen if you would only use JUnit 4 and run tests from IDEA via Gradle.

Yes, I understand that. We'd really have to migrate away from JUnit period. Which would really not be cool as I have invested a lot of time writing various JUnit 5 extensions

[marcphilipp]

Thanks for opening the issue. I tried removing the dependency to hibernate-testing:test. I can now run test for the various sub-projects, but not the database specific tests (matix_derby, etc.). I'll do some more investigating

Have you tried replacing each with configureEach as I suggested above?

[marcphilipp]

Yes, I understand that. We'd really have to migrate away from JUnit period. Which would really not be cool as I have invested a lot of time writing various JUnit 5 extensions

That sounds drastic. I don't quite follow your reasoning. What would you migrate to? The IDEA issue I reported applies to any testing framework supported by Gradle, i.e. vanilla JUnit 4, TestNG, and JUnit Platform.

[sebersole]

I thought the problem was in the combination of IntelliJ and JUnit, but you are saying that the problem happens regardless of the actual test platform used - JUnit, TestNG.

…

On Sat, May 15, 2021 at 10:45 AM Marc Philipp ***@***.***> wrote: Yes, I understand that. We'd really have to migrate away from JUnit period. Which would really not be cool as I have invested a lot of time writing various JUnit 5 extensions That sounds drastic. I don't quite follow your reasoning. What would you migrate to? The IDEA issue I reported applies to any testing framework supported by Gradle, i.e. vanilla JUnit 4, TestNG, and JUnit Platform. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2615 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABZIEZPXYKJYH6K453QJ6DTN2JIPANCNFSM444FVFHQ> .