flutter test --coverage and flutter test --watch

[Hixie]

• flutter test --coverage should write:
  • two identical files, a base file and a coverage file,
  • plus a file that lists the last modification times of all the test files.
• flutter test --watch should watch the file system of the current package.
  • If a file under lib/ changes, it should rerun all the tests and create a new base file and timestamp file.
  • If a file under 'test/' changes, it should rerun all the tests with timestamps that indicate the file changed since the base file was written.
  • after flutter test --watch runs some tests, it should merge the new coverage with the base file and overwrite the coverage file.
• We should document the caveats here:
  • Tests that have expectations or that depend on non-test files won't be run when those files change.
  • Reducing coverage won't be noticed.
  • Coverage is per-line and hitting any expression on the line counts as covering the line.

[Hixie]

cc @aam

[aam]

implementing flutter test --watch might be the shortest path to better "single test run" performance since that would provide natural lifecycle boundary for warm compiler.

cc @a-siva

[aam]

cc @keertip @mraleph

[DanTup]

I posted a comment on #15767 but it's relevant here. If this is our way of providing better test performance, it'd be great if this has some daemon mode where we can run it and use a JSON API to run individual tests. That way the user can use the existing functionality in the editor (eg. the Run/Debug links embedded in the editor next to tests, or the Test Runner tree) and get faster performance without having to have all tests run on every change.

[apaatsio]

• If a file under lib/ changes, it should rerun all the tests and create a new base file and timestamp file.

May I suggest a slight change. I would like to see a test runner that reruns only the tests that are affected by the file change, not all tests. This has a notable impact on the developer experience since it shortens the feedback loop significantly, especially on larger projects.

Jest works this way when you use --watch. It also offers an option to run all tests at every code change with --watchAll.

[zoechi]

#27192 seems to target the --watch part.

[dnfield]

#27192 likely has a lot of what is needed, but appears to have been abandoned. Anyone interested in working on this feature should probably start with the work done in that PR though.

[szotp]

It looks like it won't be implemented anytime soon so I'm considering a workaround - since flutter already supports hot reload, I could add my tests as a package directly to the app as dev dependency, and re run tests on each hot reload.

EDIT: This actually works pretty well, I've made a package for other people to try it: https://github.com/szotp/hottie/

[HerrNiklasRaab]

Related: Dart-Code/Dart-Code#2878

[danielgomezrico]

Is there any possibility that flutter test --watch run only the tests related to the file that changed?
I loved the behavior of https://github.com/guard/guard-test.

[YazeedAlKhalaf]

can we possible have something similar to how bazel does caching of tests.

if you're familiar with running a project that has bazel, you will notice that only tests of files changed are run after the first run which runs everything.

if we integrate this into the watch flag it would make development so much faster.