Test output is hard to read

[dnfield]

The compact reporter isn't really compact if you have errors, info messages, or skips in your test run. It's particularly difficult if there are multiple failures, or if you're using CI and the failures don't happen to be the very last tests that are run.

I have started work on addressing this: master...dnfield:test_reporter but want to solicit feedback. Some things I can think of that we want to acheive:

1. Any invocation of the test package use the new reporter (whether for packags/flutter, packages/flutter_tools, or wherever else in our repo). My WIP change is only handling flutter test at the moment.
2. Right now I'm just wrapping the json reporter. We could write our own reporter, but ultimately I'd like to see us able to use the data from our test runs in a more structured manner for analytics, which will probably require wrapping the JSON reporter anyway.

/cc @jonahwilliams

[dnfield]

I'm starting to think that it might not be entirely reasonable to solve this in a single way.

It would probably be easier to just make a new reporter for test (compact-summary?) to solve this issue, and just wrap the json reporter using similar logic to what's posted here. We could look at using the JSON reporter for CI - although there's something in the flutter test logic where we disallow using botht hat and --coverage, which may not work.

[hampsterx]

Compared to pytest (python test runner) the flutter test command is quite painful/limited and using it does not inspire joy with our team.

Eg no ability to add custom arguments to add new behaviour or halt on first test failure for example.

 flutter test | grep -E "\[E]";

This is our sad solution to run our test suite and only report on errors, when you have a large test suite you really do not care to see passing tests (a dot "." is sufficient) it just becomes a chore everytime to scroll up and hunt for when "-1" first occurred to get to the first fail.

[natebosch]

halt on first test failure

Latest versions have a --fail-fast argument.

only report on errors

I'll add a --report=failures-only for this. Tracking at dart-lang/test#829

[gnprice]

It looks like --reporter failures-only has landed in package:test — thanks @natebosch!

I've sent #148739 to accept the same argument in flutter test and pass it through. (Also --reporter silent, which it looks like we didn't have either.)

Once that's in, @dnfield would you like to enable that option in CI? Are there other changes you'd want to see, either as blockers for using failures-only in CI or as part of considering this issue solved?

[gnprice]

And #148739 is now merged, so --reporter failures-only is now an option in flutter test.

@andrewkolos has already filed #149611 to consider using it in CI for tests of the flutter tool itself. We could also use it in other tests in CI.