Undocumented breaking changes of `test --formatter`

[IvanIvanoff]

Environment

• Elixir & Erlang/OTP versions (elixir --version): Elixir 1.12.2 Erlang/OTP 24
• Operating system: macOS BigSur Version 11.4

Current behavior

I have a module TestFormatter.FailedTestFormatter that is used like this:

mix test --formatter Sanbase.FailedTestFormatter --formatter ExUnit.CLIFormatter --slowest 20

Its purpose is to print the failed tests at the end of the test run. It is useful when running the tests in CI and the output is very long. This way there is no need for scrolling up and down to find the failed tests as for each of them the file and line are printed at the end.

After I upgraded Elixir to 1.12 this formatter stopped working. The reason is that the messages the GenServer now receives are different.

Up to Elixir 1.11.3 the following message was received: {:suite_finished, 164457, nil}.
Since Elixir 1.12.0 the tuple has 2 elements instead of 3: {:suite_finished, %{async: nil, load: nil, run: 40905}}

Here is a repo that demonstrates the issue: https://github.com/IvanIvanoff/test_formatter_bug

I think the change was introduced here and it was intentional as the documentation for the message was updated, too: 045e3b4

Expected behavior

I looked at the changelog but I was not able to find this change explicitly listed. The change that introduces this is listed as:
[ExUnit] Print how much time is spent on async vs sync tests
I expected to have an announcement that this is an upcoming change. As we're on version 1.12.2 it is late to revert this and the issue is very easy to fix. I think updating the changelog of 1.12.0 would be the best option here so other people that upgrade to 1.12 know of this change and prepare the required changes in their code.

[josevalim]

You are 100% correct. This was an accidental breaking change, sorry about that. I will document it.

[josevalim]

Documented, thanks!