`--warnings-as-errors` can't be applied to `test_helper.exs`

[novaugust]

Elixir and Erlang/OTP versions

Erlang/OTP 27 [erts-15.2.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Elixir 1.18.2 (compiled with Erlang/OTP 27)

Operating system

macos

Current behavior

When test_helper.exs has a compilation warning, neither mix test --warnings-as-errors nor mix compile --warnings-as-errors converts the warning to an error

Looking for similar reports on this, it's my understanding that these two flags are meant to be consistent with each other.

(I'd originally stumbled over this issue in the hopes of finding a way to fail on the "won't be loaded" message, but it seems orthogonal to the writeup below which has just become "test_helper.exs isn't getting warnings-as-errors applied". Dang :))

I realize mix test --warnings-as-errors declares that it only covers test files themselves, but since mix compile also doesn't check test_helper.exs we're left with a singular file that can pass CI with warnings when no other file can (well, apart from the won't be loaded message).

As it stands, it seems that the only way to prevent passing CI with warnings is to write an additional step that greps test output for the string warning:

Example

Given a new elixir project with the following test_helper.exs

# test/test_helper.exs
List.zip []
ExUnit.start()

> MIX_ENV=test mix compile --warnings-as-errors
Compiling 1 file (.ex)
Generated my_app app
> echo $?
0
> MIX_ENV=test mix test --warnings-as-errors
    warning: List.zip/1 is deprecated. Use Enum.zip/1 instead
    │
  1 │ List.zip []
    │      ~
    │
    └─ test/test_helper.exs:1:6

Running ExUnit with seed: 690827, max_cases: 20

.
Finished in 0.01 seconds (0.00s async, 0.01s sync)
1 test, 0 failures
> echo $?
0

Expected behavior

I'd expect failures in test_helper to happen the same as when the warnings are in test files themselves, like:

> mix test --warnings-as-errors
Running ExUnit with seed: 196152, max_cases: 20

    warning: unused alias Bar
    │
  4 │   alias Foo.Bar
    │   ~
    │
    └─ test/my_app_test.exs:4:3

.
Finished in 0.01 seconds (0.00s async, 0.01s sync)
1 test, 0 failures

ERROR! Test suite aborted after successful execution due to warnings while using the --warnings-as-errors option

It would also be lovely if we could fail on warnings unique to the test context, like ..test.ex does not match "*_test.exs" and won't be loaded without having to write custom scripts to check the output, but like I said at the top, that ended up not being immediately related to the writeup reported here ;)