Test UI runs each test separately

[lukaszsamson]

Currently (1.14) elixir does not have any easy way of tracking test progress and getting test results programmatically. We can experiment with a custom formatter that outputs JSON or streams jstest compatible events to stdout

[Coffei]

You can run mix test --trace, which outputs a good amount of information, although formatted for humans.

[lukaszsamson]

That's a nogo. You'd need to parse stdout and tests can pollute it.

[Coffei]

True. We've been also using JUnit formatter to interface ExUnit with Gitlab/Bitbucket CI systems (https://hexdocs.pm/junit_formatter/readme.html). That would probably be better, but I have no idea how to plug it into an existing project when running tests. Also, there's an alternative VSCode plugin for running tests - Elixir test explorer, but it seems it is using the --trace method.

[lukaszsamson]

I know and used JUnit formatter. The downside is it's not trivial to setup and it cannot stream results in realtime

[felipeelias]

@lukaszsamson has any work done related to this? I'm looking for ways to contribute and the test runner is one of the areas that interest me.

The downside is it's not trivial to setup and it cannot stream results in realtime

What would be the most appropriate way to make this work? A custom formatter I believe, but in which format and what do you mean by "realtime"

[lukaszsamson]

I haven’t done anything concrete yet. I have a custom mix task that returns exactly the same test files that mix test does. I plan to replace the typescript implementation with that. On the test runner side I can think of two ways. One is custom formatter that streams results in json to stdout and typescript parser. The second and IMO easier is changing the test runner to make it run tests in debug adapter with noDebug option and add an exunit formatter that emits results as DAP output events. I implemented noDebug in 0.17 and added support for intercepting custom DAP messages on the typescript side.
The biggest problem with the first approach is injecting the custom formatter to client project. In case of the second one we run the code via debug adapter and can inject our code and configuration relatively easily

[felipeelias]

One is custom formatter that streams results in json to stdout and typescript parser.
The biggest problem with the first approach is injecting the custom formatter to client project.

From your comment, I understand that the challenge here is to include this custom formatter as a dependency, right?

[omh]

👋 Would it be possible, while we wait for a better solution, to just run all the tests in one go and report the result once all tests finish? I think in many cases that will be waaay faster than running each test individually. Especially on bigger projects where the start up time can take some seconds.

[lukaszsamson]

That's an option but then we loose info which test failed. It's either all red or green