Fixed 'NoTraceFileGenerated' error in simulation tests when fdbserver process could not start

[oleg68]

Problem statement

When I run FoundationDb with joshua testing framework, all tests are failed with NoTraceFileGenerated error. I asked for a help in https://forums.foundationdb.org/t/simulation-testing-of-foundationdb/2654/6 but nobody could troubleshoot it.

Finding

After some debugging I found that TestHarness.exe tried to start fdbserver but it couldn't start (in my case due an incompatible glibc version). In this case fdbserver did not write any trace files, so TestHarness.exe raised NoTraceFileGenerated. But such troubleshooting is not possible without debugging.

Solution proposal

If fdbserver failed to start, then log the root cause instead of NoTraceFileGenerated.

Testing result

After running correctness tests with changed TestHarness.exe, I receive a log with the real errors

python3 -m joshua.joshua -C /home/oleg/work/fdb/devops/clusters/joshua/fdb.cluster tail 20210713-155151-oleg-dcf3a9dcae3f1d86
[oleg@oleg2 FdbJoshua]$ python3 -m joshua.joshua -C /home/oleg/work/fdb/devops/clusters/joshua/fdb.cluster tail 20210714-104138-oleg-2602963f4dd0da38
Results for test ensemble: 20210714-104138-oleg-2602963f4dd0da38
0x100000026b446022e0000 102 c271ae5cff9c 8947850131561212810 '<Test TestUID="29080ad5-610a-4d62-8180-874852535ea3" Severity="40" Passed="0" Failed="-1" PeakMemory="0" OK="false"><StdErrOutput Severity="40" Output="/var/joshua/ensembles/20210714-104138-oleg-2602963f4dd0da38/bin/fdbserver: /lib64/libm.so.6: version `GLIBC_2.29\' not found (required by /var/joshua/ensembles/20210714-104138-oleg-2602963f4dd0da38/bin/fdbserver)" /><StdErrOutput Severity="40" Output="/var/joshua/ensembles/20210714-104138-oleg-2602963f4dd0da38/bin/fdbserver: /lib64/libpthread.so.0: version `GLIBC_2.30\' not found (required by /var/joshua/ensembles/20210714-104138-oleg-2602963f4dd0da38/bin/fdbserver)" /><StdErrOutput Severity="40" Output="/var/joshua/ensembles/20210714-104138-oleg-2602963f4dd0da38/bin/fdbserver: /lib64/libc.so.6: version `GLIBC_2.18\' not found (required by /var/joshua/ensembles/20210714-104138-oleg-2602963f4dd0da38/bin/fdbserver)" /><StdErrOutput Severity="40" Output="/var/joshua/ensembles/20210714-104138-oleg-2602963f4dd0da38/bin/fdbserver: /lib64/libc.so.6: version `GLIBC_2.32\' not found (required by /var/joshua/ensembles/20210714-104138-oleg-2602963f4dd0da38/bin/fdbserver)" /><StdErrOutput Severity="40" Output="nsembles/20210714-104138-oleg-2602963f4dd0da38/bin/fdbserver)..." /><StdErrOutputTruncated Severity="40" BytesRemaining="272" /><ExitCode Code="1" Severity="40" /><TestUnexpectedlyNotFinished /></Test>\n'
...

Code-Reviewer Section

Please check each of the following things and check all boxes before accepting a PR.

• The PR has a description, explaining both the problem and the solution.
• The description mentions which forms of testing were done and the testing seems reasonable.
• Every function/class/actor that was touched is reasonably well documented.

If this PR is made against a release-branch, please also check the following:

• This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or master if this is the youngest branch)
  • Fixed 'NoTraceFileGenerated' error in simulation tests when the fdbserver process could not start #5177
  • Fixed 'NoTraceFileGenerated' error in simulation tests when fdbserver process could not start #5178
• There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)
  • The change is small
  • Testing is quite importand especially against release branches

[foundationdb-ci]

AWS CodeBuild CI Report

• CodeBuild project: foundationdb-pull-request-build
• Commit ID: 0bd01f6
• Result: FAILED
• Build Logs (available for 7 days)

[foundationdb-ci]

AWS CodeBuild CI Report

• CodeBuild project: foundationdb-pull-request-build
• Commit ID: 04f70db
• Result: SUCCEEDED
• Build Logs (available for 7 days)