Flank Java failed with Exit Code 255 even though all tests passed.

[doodla]

I triggered the tests via CircleCI.

All the tests passed without any failures.

jUnit Aggregated Report -

Flank Output -

[rciovati]

Could you confirm that all the junit xml reports contain the tests marked as passed? Under some circumstances FTL generates an empty but valid junit report (0 tests passed, 0 failed, 0 errored, etc) but the execution is actually failed. This could happen for example when the execution terminates because it exceeds the timeout or the instrumentation process crashes at startup. You should be able to check whether all the executions were successful or not just checking in the FTL dashboard.

[bootstraponline]

update on exit code numbers:

Exit code	Number
System.exit(-1)	255
System.exit(-2)	254

[runningcode]

Are you sure that there weren't any "Infrastructure failures" at the top of the log files? If a test fails to start it is counted as a failure. I've seen this in my log files as well, where every test passed, but at the top of the file there were infrastructure failures that caused some tests to not start.

[rciovati]

Anyway, I am not sure what are we supposed to do here. @bootstraponline I'd vote for just close the issue.

[bootstraponline]

I think the fix is the same as #161 meaning we'd want to check the matrix status to determine success/failure. Also positive exit codes would make it easier to debug the code.

[sridhar-vadlamani-ck]

@runningcode There were no infrastructure failures or test failures. I visited each matrix on the console and checked.

[runningcode]

They might not appear in the console if, for example, test lab returns a 503 service overloaded error.

[rciovati]

I think the fix is the same as #161 meaning we'd want to check the matrix status to determine success/failure.

Flank doesn't check the junit reports to determine the final run status but it parses each shard's command line output.

Also positive exit codes would make it easier to debug the code.

Not sure if it is worth but we can change that. Feel free to suggest any specific exit code that you think it makes more sense to return 🙂

With regard to this specific issue, unless we have more information about what went wrong (i.e. command line debug logs, junit reports, etc) I think there isn't much we can do about it.

[bootstraponline]

Flank doesn't check the junit reports to determine the final run status but it parses each shard's command line output.

Right, I'm saying that's the problem. API based checking will be the most reliable solution. Assuming we get the matrix id from the CLI then checking should be trivial.

There's a whole category of issues that'll be solved by dropping the gcloud Python CLI dependency.

[bootstraponline]

Hi, thanks for opening an issue on the Flank repo. The Kotlin rewrite is now available for testing and feedback. This replaces the Java version of Flank.

• https://github.com/TestArmada/flank/releases/tag/flank_snapshot

Join us in the#flank channel on the Firebase slack to chat. If this is still an issue in the new release, comment and let me know.

• https://firebase.community/