twister: Twister cannot properly handle runners errors (flashing)

[PerMac]

Describe the bug
Errors occurring during flashing devices are not handled properly. There are several scenarios related to this issue.
I am able to reliably reproduce it on nrf5340 board, which raises errors during programming when app protected is not disabled, however, I see the same issues occasionally on other boards when an error occurs during programming.
Scenarios:
1.) configuration: when '--ninja' is used (both with and without --hardware-map): the test fails as soon as there is an error (which is good). However, the fail is reported as FAILED Cmake build failure (device) and treated as build error. Therefore, using --retry-failed does not retries such failures (which is bad).

• expected behavior: Such fail is not treated as a build error and the failed test is retried when --retry-failed is used

2.) configuration: without --ninja and without --hardware-map: the test ends only after timeout, even though "make" reports earlier there was a fail (Makefile:748: recipe for target 'flash' failed). This adds ~60s of wasted time to the test suite which can stack to some meaningful time in big test suites. In this scenario, --retry-failed works as it should

• expected behavior: test should be terminated as soon as there is an error in make for flash.

3.) configuration: without --ninja and with --hardware-map: Similar to 2.). This time, the output about make flash failing is not printed out. The rest is the same. Tests can be retried but are terminated only by timeouts.

• expected behavior: test should be terminated as soon as there is an error in make for flash.

The fact that hardware-related errors are seen as build errors results in printing out/saving in the report a lot of useless information which hinders the real issues. In such a case the full build log is dumped, even though the building was fine.

Impact
Can significantly increase the execution time for big test suites. Failed tests might not get retried. Hinders real errors with full build logs.

Environment (please complete the following information):

• OS Ubuntu: 18.04
• Toolchain: Zephyr SDK
• Commit SHA or Version used: zephyr-v2.5.0-276-g8e75fff5808f

Additional context
@nashif I also wonder if this could be the cause for "log mixing" problem #25719. I saw in my setups, that when flashing failed, the already programmed app can be executed giving not matching output.

[nashif]

Those are multiple issues, it would be nicer to open 1 issue for each problem with more details.

[nashif]

configuration: when '--ninja' is used (both with and without --hardware-map): the test fails as soon as there is an error (which is good). However, the fail is reported as FAILED Cmake build failure (device) and treated as build error. Therefore, using --retry-failed does not retries such failures (which is bad).

We do not retry cmake/build failures and this is by design. Can you give more details about the failure? Why exactly does it fail?

[PerMac]

@nashif I don't think splitting this issue would help, as all of the issues (most likely) have the same root cause, i.e. flashing a board being a part of the building stage.
There can be various reasons for failures during flashing, e.g some connection issues or, like in the case where I can have the error all the time, when the board is "protected" and needs "recovery" before being flashable (I guess this is specific to nrf5340 board).
I think that having flashing as a separate procedure (not part of the build stage) might solve these issues.
Anyway, I will try to debug the reported issues further and if I see that any of them can be solved on its own, I will descope it from here and create a separate bug report.