failfast option should be available also for error

[richtja]

Based on our documentation, the --failfast option interrupts the job on first fail. And this is working as expected. But IMO this shouldn't be limited only to failure, but to errors as well. Because when you don't want to run next tests after failure, there is no difference between fail and error.

This example shows the current behavior.

$ avocado run --nrunner-max-parallel-tasks=1 --failfast ../avocado/examples/tests/errortest.py /bin/true /bin/false ../avocado/examples/tests/passtest.py 
JOB ID     : 868593a1d9855d5ceaaa83042df0042d2f96df90
JOB LOG    : /home/jarichte/avocado/job-results/job-2022-04-07T11.40-868593a/job.log
 (1/4) ../avocado/examples/tests/errortest.py:ErrorTest.test: STARTED
 (1/4) ../avocado/examples/tests/errortest.py:ErrorTest.test: ERROR: This should end with ERROR. (0.04 s)
 (2/4) /bin/true: STARTED
 (2/4) /bin/true: PASS (0.01 s)
 (3/4) /bin/false: STARTED
 (3/4) /bin/false: FAIL (0.01 s)
Interrupting job (failfast).
RESULTS    : PASS 1 | ERROR 1 | FAIL 1 | SKIP 1 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/jarichte/avocado/job-results/job-2022-04-07T11.40-868593a/results.html
JOB TIME   : 2.45 s

Test summary:
../avocado/examples/tests/errortest.py:ErrorTest.test: ERROR
/bin/false: FAIL

IMO this should be the results:

$ avocado run --nrunner-max-parallel-tasks=1 --failfast ../avocado/examples/tests/errortest.py /bin/true /bin/false ../avocado/examples/tests/passtest.py 
JOB ID     : 868593a1d9855d5ceaaa83042df0042d2f96df90
JOB LOG    : /home/jarichte/avocado/job-results/job-2022-04-07T11.40-868593a/job.log
 (1/4) ../avocado/examples/tests/errortest.py:ErrorTest.test: STARTED
 (1/4) ../avocado/examples/tests/errortest.py:ErrorTest.test: ERROR: This should end with ERROR. (0.04 s)
Interrupting job (failfast).
RESULTS    : PASS 0 | ERROR 1 | FAIL 0 | SKIP 3 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/jarichte/avocado/job-results/job-2022-04-07T11.40-868593a/results.html
JOB TIME   : 2.45 s

Test summary:
../avocado/examples/tests/errortest.py:ErrorTest.test: ERROR

Because the --failfast option is an old feature which was available also on legacy runner and its behavior was same from the beginning, I would like to know other opinions about this topic.

[ana]

I'd expect --failfast to fail with errors as well. I'm OK with this change.

[clebergnu]

Agreed. In fact, for any "NOT OK" status, currently meaning ERROR, FAIL, INTERRUPTED.

[richtja]

@ana @clebergnu thanks for your replies, because this discussion is here for 20 day without any replies which would be against this change. I created #5357, and I will start working on it.