`ue4 test` doesn't exit with non-zero with failed tests

[TBBle]

We recently noticed that we had passing builds with failing tests.

It turns out we were relying on ue4 test as the last command in our pipeline, and it fails to detect the test failures.

We do see this:

[2020.03.10-03.56.20:514][204]LogAutomationCommandLine: Display: Setting GIsCriticalError due to test failures (will cause non-zero exit code).

so UE4 is exiting with error code 1.

None of the strings checked in automationTests appear in our logs though.

The most-likely candidate for a string-based check looks like:

[2020.03.10-03.56.00:819][ 32]LogAutomationController: Error: Test Completed. Result={Failed} Name={MatchmakingTest} Path={Project.Functional Tests./PluginSystem/Tests/TestLevel.OneTest}

This is with the ue4-full Docker image for 4.24.2, so it's possible the output format has changed in recent UE4 versions. I don't know off-hand if this was also failing in 4.22.2, which we moved from a couple of weeks ago.

It would probably be nicer if this (and in fact all command wrappers) propagated a failure error code back to the caller, and then we could just use that.

[adamrehn]

Earlier versions of the Unreal Engine would yield an exit code of zero in the event of an automation test failure, which is why the string checking is performed to detect failures. I'll need to run some tests with various versions of UE4 to determine the minimum version that provides accurate exit codes for propagation.

[adamrehn]

So after running tests with 4.19 through to 4.24, it actually looks like 4.19 was the last version to yield weird exit codes (e.g. returning exit code 3 after all automation tests passed successfully.) Newer versions do still appear to produce an exit code of zero if no tests are run due to an error in the logic that specifies them (e.g. an invalid filter was specified), so I've retained the existing heuristics with the addition of propagating the exit code for 4.20 and newer when the heuristics don't detect a failure. I've implemented this in commit f82aee3, which is live in ue4cli version 0.0.43, so could you please try this out and let me know if it exhibits the correct behaviour when run with your failing automation test?

[TBBle]

I think this has actually broken successful runs. The new ue4cli was just pulled into a CI job by pip, and even though all the tests passed, it failed the build.

Logs:

[2020.03.18-05.58.45:137][722]LogAutomationCommandLine: Forcing shutdown.
[2020.03.18-05.58.45:147][722]LogAutomationCommandLine: Forcing shutdown.
[2020.03.18-05.58.45:147][722]LogAutomationCommandLine: Display: **** TEST COMPLETE. EXIT CODE: 0 ****
[2020.03.18-05.58.45:159][722]LogAutomationCommandLine: Display: **** TEST COMPLETE. EXIT CODE: 0 ****
[2020.03.18-05.58.45:159][722]LogCore: FUnixPlatformMisc::RequestExit(1)

That suggests we just hit https://github.com/EpicGames/UnrealEngine/blob/4.24.3-release/Engine/Source/Runtime/Core/Private/Unix/UnixPlatformMisc.cpp#L351 which exits with return code 1 because it was a forced exit.

This is possibly a UE4 bug, that it only calls RequestExitWithStatus for failed tests, but not for successful tests, but I haven't tried to trace it down.

[adamrehn]

Ah damn, so it's different under different platforms then. I did my testing under Windows, which yields an exit code of zero under those circumstances. Given that there's no consistency across platforms, I'm going to revert the code that propagates the exit code and revert to detecting failures purely with text-based heuristics.

[adamrehn]

Right, I've switched to using a regex to detect and propagate stated exit codes in commit 08cf938, which is live in ue4cli version 0.0.44. Give that a try and let me know how you go.

[TBBle]

I didn't try a known-failing test, but the Linux auto-test run without failures now passes CI with 0.0.44, thank you.