tycho-surefire-plugin:integration-test does not execute any test

[LorenzoBettini]

I was trying the goal tycho-surefire-plugin:integration-test in an eclipse-plugin project, without any specific configuration, sticking to the default (*.IT.java), but when the goal runs in the integration-test phase no test is executed. I was trying that on a small project of mine, but I also tried the official example at

https://github.com/eclipse/tycho/tree/master/tycho-its/projects/surefire.combinedtests/bundle.test

I took the whole project and replaced a few values in the POM:

<repositories>
	<repository>
		<id>platform</id>
		<url>https://download.eclipse.org/releases/2021-12</url>
		<layout>p2</layout>
	</repository>
</repositories>

and

<build>
	<plugins>
		<plugin>
			<groupId>org.eclipse.tycho</groupId>
			<artifactId>tycho-maven-plugin</artifactId>
			<version>2.6.0</version>
			<extensions>true</extensions>
		</plugin>

and ran mvn verify.

This is (part of) the result:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running bundle.test.SubtractorTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s - in bundle.test.SubtractorTest
[INFO] Running bundle.test.AdderTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in bundle.test.AdderTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] --- tycho-packaging-plugin:2.6.0:package-plugin (default-package-plugin) @ bundle.test ---
[INFO] Building jar:...surefire.combinedtests/bundle.test/target/bundle.test-1.0.0.jar
[INFO] 
[INFO] --- tycho-p2-plugin:2.6.0:p2-metadata-default (default-p2-metadata-default) @ bundle.test ---
[INFO] 
[INFO] --- tycho-surefire-plugin:2.6.0:integration-test (default-integration-test) @ bundle.test ---
[INFO] Executing Test Runtime with timeout 0, logs (if any) will be placed at: ...surefire.combinedtests/bundle.test/target/work/data/.metadata/.log
[INFO] Command line:
        [...]

Results:

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO] No tests found

So the JUnit tests with maven-surefire are correctly executed, but no JUnit test is executed by tycho-surefire

I also tried with Tycho 2.4.0 (when this feature was introduced by @laeubi ) but the problem is still there.

Using -X does not provide any further information.

Am I missing something?

[LorenzoBettini]

By the way, I seem to understand that this feature is tested in https://github.com/eclipse/tycho/blob/e4e95b6b19a54a98ba3b0e9edbb1e8aeb3d9925b/tycho-its/src/test/java/org/eclipse/tycho/test/surefire/TestsInBundleTest.java but in that test the integration-test phase is never run, so that test does not actually verify the integration-test goal...

[laeubi]

@LorenzoBettini thanks for looking into this it has been a while since I added this and it seems I removed some tests afterwards with commit 67dfb7f but I can't remember what was the issue it seems at least only one of them should have been removed. I'm still marking this as a regression even though it seems it never worked in any released version.

[laeubi]

@LorenzoBettini the problem is that somehow the surefireprocess gets confused with the classloading and thus the junit provider does not find the annotated classes :-\

I already tried to set the dependecy to the ones used in tycho but without success.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.22.2</version>
  <dependencies>
	  <dependency>
		  <groupId>org.apache.maven.surefire</groupId>
		  <artifactId>surefire-junit47</artifactId>
		  <version>2.22.2</version>
	  </dependency>
  </dependencies>
</plugin>

[LorenzoBettini]

@laeubi so just reverting that commit does not fix things?

[laeubi]

This was just an experiment but don't work quite well as it relies on "additional.bundles" that has a different meaning in PDE. I think I know what the problem might be here but will need to take a deeper look at it tomorrow.

[LorenzoBettini]

The problem is that, if you don't rely on additional.bundles, the required bundles for testing will end up in the main code transitive dependencies, if I understand correctly...

[laeubi]

additional.bundles is inherently broken but as you said could be used for such purpose, but it is just a side-effect that it actually works.

The correct way would be adding extra library entries with scope=test but full support for this is currently work in progress.

[LorenzoBettini]

I thought that additional bundles was fixed a few releases ago. In particular, it now works as expected in Xtext

[laeubi]

I thought that additional bundles was fixed a few releases ago.

We added support for a similar side-effect as in PDE in tycho yes...

it now works as expected in Xtext

Additional Bundles are meant for that you can add some bundles in advance and PDE adds import-package/require bundle when you run a product. I think that's not what xtext want nor how it is used, and I think not how one wan't tycho to behave.

As a side-effect, this also adds the bundle and its dependencies to the classpath and as long as you only use it in test-code has a similar effect to a feature I requested for PDE here but was rejected (I implemented a POC here) and where I have now requested a slightly variant here now (but now response so far).

So it may "work" under certain circumstances, but is still broken from my point of view.

[laeubi]

I fixed the integration test + code, it should all work as desired now.

[LorenzoBettini]

@laeubi sorry for the delay: I seem to understand that:

• this only works if the current plugin is NOT a singleton
• it does not work with JUnit 5, or I wasn't able to make it work: I always get

org.apache.maven.surefire.util.SurefireReflectionException: java.util.ServiceConfigurationError: org.junit.platform.engine.TestEngine: org.junit.jupiter.engine.JupiterTestEngine not a subtype
	at org.apache.maven.surefire.util.ReflectionUtils.instantiateOneArg(ReflectionUtils.java:139)
	at org.apache.maven.surefire.booter.SurefireReflector.instantiateProvider(SurefireReflector.java:265)

in the log when tycho-surefire-plugin:2.7.1:integration-test (default-integration-test) is executed