Failure in basic-build-tests.sh isn't properly reported

[yanesca]

Failures in basic-build-tests.sh should make the CI report the failure (making the overall run red).

See for example:
https://jenkins-mbedtls.oss.arm.com/blue/organizations/jenkins/mbedtls-release-new/detail/mbedtls-release-new/854/pipeline/14892/

The CI is green and passes, but there are failures in the code coverage job.

[gilles-peskine-arm]

In https://jenkins-mbedtls.oss.arm.com/blue/organizations/jenkins/mbedtls-release-new/detail/mbedtls-release-new/854/pipeline/14892/ basic-build-test.sh reports success, so this doesn't show a problem in the code that's in the mbedtls-test repository.

Checking on basic-build-test.sh, ever since its creation on 21ab9d7, it never considered a test case failure to be a cause for failure. Only build failures and problems in creating the report were considered to be failure conditions, and this was clearly a deliberate design choice. A test case failure leads to a nonzero number in the “Total Failed:” line in the report, and a successful exit status. A failure in a test script (e.g. ssl-opt.sh errors out without running any tests) also leads to a success of the script as a whole, but this was likely unintended.

I tend to agree that this is not what we need.

A quick fix would be to exit with status 1 if TOTAL_FAIL is nonzero. This would only capture failed test cases, not failures of an auxiliary test script.

A bit of technical explanation for non-experts in the sh language: the script uses set -e, which causes it to exit if any command fails. But a pipeline such as sh ssl-opt.sh | tee … has the status of its right-hand side; the status left-hand side is ignored. We can change #!/bin/sh to #!/usr/bin/env bash and add set -o pipefail to abort the script if the left-hand side of a pipe fails, but this means the script will stop at the first failure. Alternatively, we can keep track of when the left-hand side of each pipe fails with a bit more work, as we do nowadays for the report generation.