mix test --warnings-as-errors and --failed flags with unexpected interaction

[polvalente]

Environment

• Elixir & Erlang/OTP versions (elixir --version):

Erlang/OTP 24 [erts-12.1.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Elixir 1.12.3 (compiled with Erlang/OTP 24)

• Operating system: Ubuntu 20.04 (but issue also occurs on CI machine with Alpine 3.14.2)

Current behavior

Running mix test --warnings-as-errors || mix test --warnings-as-errors --failed can cause warnings to fall through the CI workflow.
This is because the --warnings-as-errors ends up not running any tests at all, which in turn causes the --failed flag on the second command to not return any files to be compiled, resulting in a false success.

For what is worth and future reference, the current local fix is to run it as: mix test --warnings-as-errors || if [[ $? = 1 ]]; then (exit 1); else mix test --warnings-as-errors --failed; fi

Expected behavior

In my opinion this is not desirable behaviour, and perhaps could be fixed by keeping track of a failure due to warnings, effectively marking all related files as failed for the next command.

Thanks in advance!

[josevalim]

In my opinion this is not desirable behaviour, and perhaps could be fixed by keeping track of a failure due to warnings, effectively marking all related files as failed for the next command.

Agreed with this as a fix.

[polvalente]

In my opinion this is not desirable behaviour, and perhaps could be fixed by keeping track of a failure due to warnings, effectively marking all related files as failed for the next command.

Agreed with this as a fix.

@josevalim Could I submit a PR on this?

[josevalim]

Sure! Btw, there is another approach on this, which is: if --warning-as-errors fail, don't save the manifest that keep --failed entries, which means you will run the same tests again. This is likely preferred to running them all...

[josevalim]

None of the approaches seem doable in practice without coupling a lot of the test code with the test runner and with the compiler. I think we should consider those to be indeed distinct actions and, if the suite fails AFTER it successfully runs, then --failed not running test correctly reflects the status of the suite. For CI, you can continue checking the exit code to decide how to act.

[Nezteb]

https://angelika.me/2024/01/08/do-not-run-mix-test-failed/ 👀