The "Test: Run All Tests" command is much slower than "Dart: Run All Tests"

[DanTup]

What I've found is that you now have to use the VSCode test command Test: Run All Tests but this is now much much slower as it seems to be launching each time for each test file. For my TDD with over 60 test files this is now a really painful experience. Hopefully this is just a bug and the intended DX is still quick and responsive.

old Dart: Run All Tests -> 2-3sec
new Test: Run All Tests -> 40-60sec

Originally posted by @lloydrichards in #3594 (comment)

[lloydrichards]

Excellent! Thanks so much for the quick response. Will this be a hot fix anytime soon or will we need to wait til v3.28.0 in Nov?

It really slows down my TDD today so if there was a quick workaround or should I just downgrade back to use the old system?

[DanTup]

There have been quite a few changes in the test runner with this fix so it'll probably only ship in the next main release. However, I can do a preview build once I've implemented a few more of these test changes (things like automatic discovery of test files/tests without having to open the files) if you're interested in using that (it would be good to have additional eyes testing it).

In the meantime, you can switch to the old test runner without downgrading the whole extension with the useVsCodeTestRunner setting. Running the tests from the command palette/CodeLens/etc. should be unaffected too - my understanding is that it's only the Run Tests button at the top of the VS Code test runner that's doing the slow thing.

[lloydrichards]

would definitely love to help!

[DanTup]

There's a beta release of v3.28.0 here that includes all the most recent test fixes, and also automatic test discovery (the tree should populate when you load the project without needing to open/run tests first):

https://github.com/Dart-Code/Dart-Code/releases/tag/v3.28.0-beta.1

[DanTup]

There were some issues in that beta, there's a new one here:

https://github.com/Dart-Code/Dart-Code/releases/tag/v3.28.0-beta.2

[lloydrichards]

I'm not sure what's been updated in the newest version but I'm running into some issues with tests again and figured I would ask in this thread first before making a new issue. Firstly is that the Dart: Run All Tests seems to have been removed so now I need to instead use Test: Run All Tests but when I try this, I get a message saying "Unable to find any test folders"

Next, I go to the Testing tab and click Run All Tests and get this message: "Integration tests and unit tests cannot be run in a single invocation. Use separate invocations of flutter test to run integration tests and unit tests."

Next, I click Debug Tests and I get the same behaviour as before where it loads a new instance for every test suite causing the tests to run for 4-5min (I have a lot more tests now)

Finally, I go to the Debug tab, select Flutter Test from the dropdown and then Start Debugging, and I get the expected results from running Dart: Run All Tests, but now when I press f5 to start the device, it instead runs my tests 😅

I'm not sure what's going on with the Dart Testing development, but it would be nice to have a workflow that I can keep the same for at least a couple of months at a time. If there was maybe some documentation on how you expect tests to be run in TDD it would be helpful

[DanTup]

@lloydrichards sorry for the bother. Moving from our own custom runner to the new built-in runner was quite a big job and introduced a few issues. I'm confident that once the remaining differences and edge case are resolved, things should be much less volatile.

Firstly is that the Dart: Run All Tests seems to have been removed so now I need to instead use Test: Run All Tests but when I try this, I get a message saying "Unable to find any test folders"

You're right that this command was removed, and Test: Run All Tests is the expected command now. However, there was an issue if you hadn't previously loaded the test runner, VS Code doesn't seem ask us to discover the tests so we produced this error. I've fixed it in #3714 and we'll now automatically discover tests if there is any request to run tests (from anywhere) and we haven't already done so. It'll ship in the next version, but I'm also hoping to make a pre-release build (using VS Code's new functionality) next week that you could switch to.

Next, I go to the Testing tab and click Run All Tests and get this message: "Integration tests and unit tests cannot be run in a single invocation. Use separate invocations of flutter test to run integration tests and unit tests."

This sounds like a limitation of flutter test that VS code isn't accounting for. Could you file a new issue about this and I'll ensure it's also fixed before the pre-release/next release. I've filed #3727 about this.

Next, I click Debug Tests and I get the same behaviour as before where it loads a new instance for every test suite causing the tests to run for 4-5min (I have a lot more tests now)

I believe this has always been the case for debugging. IIRC, it's only when running without a debugger that we're able to run multiple suites in a single session.

Finally, I go to the Debug tab, select Flutter Test from the dropdown and then Start Debugging, and I get the expected results from running Dart: Run All Tests, but now when I press f5 to start the device, it instead runs my tests 😅

I don't quite understand here. If you're selecing a debug configuration from the debig sidebar, pressing F5 will run that config, so if you select a test config and press F5, running tests is expected behaviour. Pressing F5 is the same as clicking the green Debug button at the top of the Debug side bar next to the debug configuration.

If there was maybe some documentation on how you expect tests to be run in TDD it would be helpful

There are no real expectations. Things should generally work the way users expect them to. If anything is not working of confusing, it's always worth filing an issue and assuming it's something that can be improved rather than you're doing something wrong :-) I can't promise everything is fixable, but often things don't work the best way just because they haven't had a lot of feedback and were based on my own (very narrow) usage :-)

[DanTup]

The fixes noted above (including a fix for running all tests when there are both unit+integration tests) were included in todays pre-release release. You can switch to this in VS Code if you'd like the fixes and/or to help test new features/fixes before they're included in a stable release:

[lloydrichards]

thanks so much! will give a shot tomorrow at work :)

[lloydrichards]

Have been testing out the Pre-Release and the clicking Run Tests in the Testing tab does now work and quickly run all tests. however when I use the keyboard shortcut (ctrl + ; A) then I get the same behaviour where it runs each test group in its own debug instance and takes 15min to complete.

Is there a different keyboard binding to run the tests from the Testing tab to the Test: Run All Tests in the command pallet

[DanTup]

@lloydrichards is it possible you're holding down ctrl when you press A too? I asked because Run All Tests and Debug All Tests have similar keybindings, and Debug will be much slower because we can't combine the runs:

[lloydrichards]

if I press Ctrl + ; A then i get:

The syntax of the command is incorrect.
Exited (255)

Web devices are not supported for integration tests yet.
Exited (1)

and if I press Ctrl + ; Ctrl + A then I get each test as its own debug instance.

I can't find any keybinding that works to run what I click in the Testing tab

[DanTup]

if I press Ctrl + ; A then i get:

The syntax of the command is incorrect.
Exited (255)

Web devices are not supported for integration tests yet.
Exited (1)

Is this different behaviour to running the Test: Run All Tests command? VS Code is handling the shortcut keys and should invoke the same command, so if those two things aren't behaving the same it seems like something is wrong in VS Code (assuming you haven't changed the keybindings).

Assuming you get the same behaviour when running the command too, the second message is perhaps correct - it's trying to run your integration tests and your selected device (in the status bar) is the Chrome device, but integration tests are not supported on web. You'll need to select another device for running the integration tests.

As for "The syntax of the command is incorrect.", that seems like a bug though I can't reproduce. Can you capture a log:

• Run the Dart: Capture Debug Logs command
• Press Ctrl + ; then A to reproduce the issue
• Click Cancel on the logging notification to stop logging and open the log

You can attach the log to a new issue in this repo describing that issue. Thanks!