"Test: Run All Tests" shows error prompt but no error output

[nickmeinhold]

Describe the bug

When I use "Test: Run All Tests" I get this error message

But I can't find the actual error output anywhere, there doesn't seem to be anything in the Debug console or the Output

I also get this, but as you can see I have a device attached

The tests do seem to run but the only indication of that is in the Debug console - the Testing tab shows 0 tests passed in the summary but then individual tests show as passing

To Reproduce

EDIT: I went into each sub-folder of the monorepo mentioned below and tried running tests until I narrowed it down to a flutter plugin and then specifically to an integration test in the web platform plugin's example project, found here:
packages/flutter/plugins/flutter_box2d/flutter_box2d_web/example/
It's a web integration test and follows the convention from https://docs.flutter.dev/cookbook/testing/integration/introduction#5b-web but I'm guessing that doesn't work with the test runner?

Steps to reproduce the behavior:
(I will try to create a minimal repro but in case this is useful in the meantime)

1. Clone https://github.com/enspyrco/monorepo
2. Set "dart.projectSearchDepth": 8 (I don't think it needs to be as high as 8 but it worked so I left it as 8)
3. Use the Test: Run All Tests command

Expected behavior
Tests show running state in the TESTING tab and all tests run.

• Operating System and version:
  MacOS 12.4

• VS Code version:
  1.68.1

• Dart extension version:
  3.42.1

• Dart/Flutter SDK version:
  Flutter 3.0.3

• Target device (if the issue relates to Flutter debugging):
  I have tried targets of macos and an iphone SE running 15.4.1

[DanTup]

So I think what's happening here is when we try to run the web integration tests, we notice the selected device is not valid for this project (only web is), so we try to prompt for a device (which makes sense when you run one project, but perhaps less sense if you've simultaneously kicking off lots of them). However, I think the device selector is immediately hidden by another project starting.

In this case, since there's only one valid device, we could probably auto-select it. However that wouldn't fix a similar issue if your project support multiple platforms and you happen to have a different one selected. Perhaps when running integration tests we should just skip any that aren't support for the selected device and show a message, and then it's up to you to run them per-device if you need to.

(Also, perhaps we should make it easier to run all non-integration tests 🤔)

[DanTup]

@nickmeinhold can you try upgrading to pre-release v3.43.20220628 and see how things work for you? Main changes are:

• Don't reject promises for VS Code when failing to run some test sessions in a multi-suite session (eg. Run All Tests)
• Don't try to prompt for devices during multi-suite test runs (we already suppressed prompting on errors, but weren't suppressing device selector)

The result is that if your selected device is not valid for a set of tests, they will be skipped. If you try to run them individually, you'll see the errors (or prompts for things like a valid device).

[nickmeinhold]

@DanTup works like a charm! Really appreciate your (typically) speedy response!

[DanTup]

Great, thanks for confirming!