[BUG] [GREEN FAILURES] host-testbench.sh ignores errors

[marc-hb]

Describe the bug
host-testbench exits with success 0 when tests fail. Random, recent example in PR #2751:

https://travis-ci.org/github/thesofproject/sof/jobs/672996210 is green but:

eqiir test failed!

I didn't even have to spend time to search for this example, I only looked at the most recent PR.

Ignoring failures is the very worst type of validation issue because it makes everyone think everything is OK when it's not.

Do NOT fix the unreported eqiir failure, fix host-testbench.sh error handling first.

From https://en.wikipedia.org/wiki/Test-driven_development#Test-driven_development_cycle

see if the new test fails [...] this rules out the possibility that the new test is flawed and will always pass.

For a start, host-testbench.sh has no set -e (set -o errexit)

$ shellcheck ./scripts/host-testbench.sh
In scripts/host-testbench.sh line 62:
if [[ $? -ne 0 ]]; then
      ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

https://github.com/koalaman/shellcheck/wiki/SC2181

One possible alternative:

set -e
...
if test1 ; then
  ...
else
  ...
  : $((failures++))
fi

exit $failures

To ignore some errors use "|| true":

  this_command_can_fail_and_it_is_OK || true

https://github.com/thesofproject/sof/pull/2657/files is a good codesyle example.

To Reproduce

./scripts/host-testbench.sh |grep -i fail
eqiir test failed!
./scripts/host-testbench.sh; echo $?
0

Expected behavior
Failures are not ignored and painted green.

Impact
Regression are never noticed.

Environment

1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
   • SOF: master branch, HEAD=9915b803f73e
2. Name of the topology file
   N/A
3. Name of the platform(s) on which the bug is observed.
   • Platform: build system

Screenshots or console output

./scripts/host-testbench.sh |grep -i fail
eqiir test failed!
./scripts/host-testbench.sh; echo $?
0

[marc-hb]

Reducing the default verbosity would also help IMHO.

[lgirdwood]

@singalsu fyi - is this a quick fix for you ?