should mix test --failed fail when there are test compilation errors

[dvic]

Elixir and Erlang/OTP versions

1.17 + OTP 26

Operating system

Mac

Current behavior

mix test || mix test --failed succeeds when a test contains a compilation error

learned through: https://angelika.me/2024/01/08/do-not-run-mix-test-failed/
affects also: karolsluszniak/ex_check#48

Expected behavior

I was expected that mix test --failed fails when there are test compilation errors.

[josevalim]

This is tricky. If this happens, we most likely want to tackle the whole suite as invalid. However, if something failed before and tests do not compile, we don’t want to run the whole suite. So perhaps, when compiling tests fail, and there is nothing in the failure manifest, we load all tests.

however, I think we had a discussion about this in the past. We may have a flag or something to deal with this?

[dvic]

Not sure about the flag, but I now do see this past discussion: #10781

What I understand from the discussion is that at one point mix test --failed was changed to run all tests if there were no previously failed ones:
ae48325

But that got reverted again? e717a4b