"Test: Run All Tests" command does not run all tests

[johnnypcc]

Describe the bug
VSCode Test Runner doesn't run all tests, some are left unrun.

To Reproduce
Steps to reproduce the behavior:

1. Go to the testing tab
2. Click on the button that runs all tests (Run tests)

Expected behavior
All tests should run

Screenshots
Added

Versions (please complete the following information):

• VS Code version: 1.60.2
• Dart extension version: v3.27.0
• Dart/Flutter SDK version:
• Flutter (Channel stable, 2.5.1, on macOS 11.6 20G165 darwin-x64, locale en-CA)
  Dart SDK version: 2.14.2 (stable) (Wed Sep 15 12:32:06 2021 +0200) on "macos_x64
  
  "

[lloydrichards]

Im getting the same issue when using the Dart: Run All Tests

What I've found is that you now have to use the VSCode test command Test: Run All Tests but this is now much much slower as it seems to be launching each time for each test file. For my TDD with over 60 test files this is now a really painful experience. Hopefully this is just a bug and the intended DX is still quick and responsive.

old Dart: Run All Tests -> 2-3sec
new Test: Run All Tests -> 40-60sec

[DanTup]

Thanks for the report! I overlooked the Test: Run All Tests command. I suspect it's running all discovered tests and therefore only runs those already visible in the tree. I think it may be fixed once #3591 is implemented, but I'll keep this open to track it to be sure.

The reason for Test: Run All Tests being slower might be that VS Code is handing us the full list of tests and they're being run one-at-a-time (or at least, one-file-at-a-time) whereas Dart: Run All Tests might be using a single test process (eg. just running dart test test). I've opened #3595 to look into that. Thanks!

[johnnypcc]

Thanks for the report! I overlooked the Test: Run All Tests command. I suspect it's running all discovered tests and therefore only runs those already visible in the tree. I think it may be fixed once #3591 is implemented, but I'll keep this open to track it to be sure.

@DanTup You're welcome. But even after it discovers all tests, and I try to run them all, they're still not run.

[johnnypcc]

this is now much much slower as it seems to be launching each time for each test file. For my TDD with over 60 test files this is now a really painful experience. Hopefully this is just a bug and the intended DX is still quick and responsive.

Yeah, it's much much slower :(

[DanTup]

after it discovers all tests, and I try to run them all, they're still not run.

What's the relative path to one of your test files from the workspace folder? Eg. if you open a test file and find it in the Explorer tree, then right-click it and choose "Copy relative path", what does that look like? Is it outside of the test folder, or is the test folder not in the root of the folder opened in VS Code?

Can you reproduce it in a newly created project?

Yeah, it's much much slower :(

As a short-term workaround, you can still use the Dart: Run All Tests command for now which should behave the same as before the update (it should find all tests and be faster). The fix in #3595 will make the "Run Tests" button in the new VS Code runner the same speed as long as you're running all tests (eg. haven't hidden any) and are running without debugging. When debugging, or selectively picking tests (eg. excluding some), we need to run them by file, so that will be slower.

[johnnypcc]

What's the relative path to one of your test files from the workspace folder?

All my tests, those which were run and those that weren't run, sit together under the test dir, which itself is in the root dir.
Thanks

[DanTup]

@johnnypcc is it the same tests that are skipped each time? Are you able to make a small sample project that triggers the issue that you can share?

A log file might be useful too (note: it will include information about your tests, so best to capture in a project that's not sensitive):

• Run the Dart: Capture Logs command from the command palette
• Untick Analysis Server but leave the others ticked and click OK
• Click the Run All Tests button, and wait for all sessions to complete (they're run sequentially)
• Click Cancel on the logging notification to stop logging and open the log file

You can send the log to me at logs@dartcode.org (do not send anything sensitive), or you could try searching through it for some of the filenames that did not run, to see if there's anything obvious there.

[DanTup]

I was able to reproduce this and figured out what it is. It happens when we run multiple test suites sequentially - the first one "ends" the test run and subsequent test results are not reported.

I have a fix for this, though I've also improved things so many test runs can run in a single debug session (which will make them much faster - #3599).

I may patch the first fix into a hotfix release, though the other one is tied in with some larger changes and will be in the next major version.