Testsuite fatal error output not shown

[molexx]

With a config like this:

testlogger {
	showStandardStreams true
	
	showPassedStandardStreams false
	showSkippedStandardStreams false
	showFailedStandardStreams true
}

if the test suite run itself fails - i.e. not 'a test does not pass' but the test execution failing due to something else e.g. an @Sql setup script with an error in it - gradle fails and stderr says:

> Task :test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.

but there's no indication as to what the problem was. If I run again with --info I can see that the stdout explains the problem in detail, but that disables the above testlogger filters.

Could testlogger notice the suite failure and show the output please?

[radarsh]

Hmm, that's not a nice behaviour from the plugin. Let me see what I can do. Thanks for reporting this.

[radarsh]

Hi @molexx, would you be able to provide an sscce for this so that I can provide an accurate fix? Thanks.

[molexx]

I've investigated what's happening: System.exit(5) is being called.

I guess this is somewhat unexpected behaviour for JUnit tests but I have seen several StackOverflow posts suggest that as a way to abort the rest of the test suite so as to not waste when a fatal error has happened.

Would it be right for gradle-test-logger-plugin to catch the VM's non-zero exit code, treat that as a failed test and show the output?

Thanks.

[radarsh]

Interesting. Would be good to find out what Gradle would do without the test logger plugin applied and with a configuration like this:

test {
    testLogging {
        showStandardStreams = true
    }
}

[radarsh]

Hi @molexx, could you share a bit more detail about this please? Where is System.exit(5) being called from? Is it in a setup method? Also what happens when you try this? Thanks. This info will be quite invaluable.

[molexx]

Behaviour is the same for @Test and @Before methods which do some output before calling System.exit(5):

Without gradle-test-logger applied and without the above test{ testLogging { showStandardStreams=true}} the stdout/stderr messages are not shown.

Without gradle-test-logger applied but with the above test{ testLogging { showStandardStreams=true}} the stdout/stderr messages are shown.

With gradle-test-logger applied the test is reported as SKIPPED (in yellow), then the expected behaviour happens: if testLogger's showStandardStreams and showSkippedStandardStreams are true the messages are seen.
I don't think it should be detected as SKIPPED though, as the test suite's VM aborted without the test completing and with status 5 (which gradle notices and reports as in the first post).

@BeforeClass methods are similar without gradle-test-logger - with test{ testLogging { showStandardStreams=true}} the outputs show.

But with gradle-test-logger applied, even with all options set to true:

testlogger {
	showStandardStreams true
	
	showPassedStandardStreams true
	showSkippedStandardStreams true
	showFailedStandardStreams true
}

the System.out and System.err in the @BeforeClass method are not seen.

[radarsh]

I don't think it should be detected as SKIPPED though, as the test suite's VM aborted without the test completing and with status 5 (which gradle notices and reports as in the first post).

Unfortunately, there is nothing the plugin can do about this as this is Gradle's internal behaviour. I have fixed the missing system out issue though. It was a genuine bug, thanks for reporting it and the detailed analysis 👍

[molexx]

Ah ok seeing the messages is the main thing, thanks! :-)