How can I make a build succeed even though a test case has failed?

[estepper]

Recently I had to upgrade the Tycho version that's used in the CDO build and now I notice that failed tests start to make the entire build FAILED (instead of marking it UNSTABLE).

I think that's the default behavior of Maven, but I've always had this option in the parent POM:

      <build>
        <plugins>
          <plugin>
            <groupId>org.eclipse.tycho</groupId>
            <artifactId>tycho-surefire-plugin</artifactId>
            <configuration>
              <testFailureIgnore>true</testFailureIgnore>

It doesn't seem to have the expected effect anymore, though.
Now I also tried the -fn command line argument, but it seems to make no difference, either.

Any ideas?

[laeubi]

If you where previously using -Dtest=.... on the root level of your build tree it might now be necessary to also include -Dsurefire.failIfNoSpecifiedTests=false as maven-surefire might otherwise complain about

No tests were executed! (Set -DfailIfNoTests=false to ignore this error.)

for your eclipse-plugin packaged project if they do not match anything (the error message is a bit misleading, this is tracked in SUREFIRE-1910).

But -fn should always work, so probably you can share the exact error message you get including a stacktrace (-e maven option)

[estepper]

Thanks for the fast reply!

I don't get how this failIfNoTests is related to my problem. It does execute a non-zero number of test cases.

Here's the result of executing with -fn -e:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  14:04 min
[INFO] Finished at: 2022-11-01T15:12:18+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on project org.eclipse.net4j.tests: There are test failures.
[ERROR] 
[ERROR] Please refer to C:\develop\cdo-master\git\cdo\plugins\org.eclipse.net4j.tests\target\surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on project org.eclipse.net4j.tests: There are test failures.

Please refer to C:\develop\cdo-master\git\cdo\plugins\org.eclipse.net4j.tests\target\surefire-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures.

Please refer to C:\develop\cdo-master\git\cdo\plugins\org.eclipse.net4j.tests\target\surefire-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
    at org.apache.maven.plugin.surefire.SurefireHelper.throwException (SurefireHelper.java:289)
    at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution (SurefireHelper.java:161)
    at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary (SurefirePlugin.java:364)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked (AbstractSurefireMojo.java:1041)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute (AbstractSurefireMojo.java:857)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[ERROR] 
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf org.eclipse.emf.cdo:org.eclipse.net4j.tests

I hope that tells you something that you can tell me then ;-)

[estepper]

I attach the entire console output file...
maven.log

[laeubi]

Please note that the error message says:

Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test

but in your pom snippet you configure tycho-surefire-plugin ...

[estepper]

Hmm, that is interesting! Now I notice that these tests have not been executed at all formerly.

Unfortunately I'm not Maven'ish enough to see how to fix the problem. Especially since all of this (without any changes in my configs) used to work for ages ;-(

If I get you right, you're suggesting that now the maven-surefire-plugin is executed instead of (or in additon to?) the tycho-surefire-plugin. If that's the case, how can it happen? I mean, who/what is starting them? And what can I do against it?

Please note that none of my POMs contains the string "maven-surefire"...

[laeubi]

Now I notice that these tests have not been executed at all formerly.

Support for sure-fire-test was added in Tycho 2.4.0 see
https://github.com/eclipse-tycho/tycho/blob/master/RELEASE_NOTES.md#execute-unit-tests-with-eclipse-plugin-packaging

I mean, who/what is starting them? And what can I do against it?

The more interesting question is why do you write test that you never want to be executed? :-)

Basically you have two choices:

1. Keep the execution, getting better test coverage, and additionally configure maven-sure-fire to not fail on failing tests: https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
2. You can also configure surefire to skip test execution: https://maven.apache.org/surefire/maven-surefire-plugin/examples/skipping-tests.html#skipping-tests

[estepper]

So the 2.4 relnotes say "From now on [...] one is able to execute unit-tests with maven-surefire" which sounds reasonable to me. I don't fully understand the motivation and the consequences because, apparently I've never had the need or desire to use maven-surefire. Again, that string does not occur in any of my POMs, and never did. From that sentence I would expect that I'd have to add some POM markup to use maven-surefire. But it appears that now it kicks in even without asking for it. Is that true?

So I understand you're suggesting to disable all maven-surefire tests with this:

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <skipTests>true</skipTests>
        </configuration>
      </plugin>

Then I get dozens of warnings like "[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-surefire-plugin is missing". Kind of makes sense, but I've never asked for maven-surefire. So I've never specified a version for it (even though it appears that currently maven-surefire-plugin:2.22.2 is used).

What version should I pick now?
And will I have to adjust it in the future so that the skipTests continues to apply??

The more interesting question is why do you write test that you never want to be executed?

Believe me, it has a good reason ;-)

[laeubi]

But it appears that now it kicks in even without asking for it. Is that true?

yes surefire is part of the so called "default life-cycle" (as it is for "standard" maven jar packaging) so tycho actualy now behaves more like maven user would expect it.

I don't fully understand the motivation and the consequences because, apparently I've never had the need or desire to use maven-surefire.

Before you mostly can only use tycho-surefire (what actually is some kind of plugin test) with a special packaging in an extra module. surefire now enhances this with a strategy where you can run real unit test, that is those not requiring OSGi at all, next to the code in the same module.

So I understand you're suggesting to disable all maven-surefire tests with this:

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <skipTests>true</skipTests>
        </configuration>
      </plugin>

Yes this skips the execution of tests.

Then I get dozens of warnings like "[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-surefire-plugin is missing". Kind of makes sense, but I've never asked for maven-surefire. So I've never specified a version for it. What version should I pick now??

Depends a bit on your setup, from the above message 2.22.2 seems suitable but actually any never version should be sufficient.

The more interesting question is why do you write test that you never want to be executed?

Believe me, it has a reason ;-)

Sure, but in most cases if one writes test you want them to be executed, so Tycho support this now :-D

[estepper]

Ok, I added another xml block under pluginManagement in the parent pom:

        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
	        <version>2.22.2</version>
        </plugin>

That seems to fix all my problems now.

The pleasure of convention over configuration has always been a mystery for me. Now, in the light of changing conventions, it looks even worse to me ;-(

But thanks for your help, anyways!