Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Test: Run All Tests" command does not run all tests #3594

Closed
johnnypcc opened this issue Oct 6, 2021 · 11 comments
Closed

"Test: Run All Tests" command does not run all tests #3594

johnnypcc opened this issue Oct 6, 2021 · 11 comments
Labels
in testing Relates to test execution of Dart/Flutter tests for end users is bug
Milestone

Comments

@johnnypcc
Copy link

Describe the bug
VSCode Test Runner doesn't run all tests, some are left unrun.

To Reproduce
Steps to reproduce the behavior:

  1. Go to the testing tab
  2. Click on the button that runs all tests (Run tests)

Expected behavior
All tests should run

Screenshots
Added

Versions (please complete the following information):

  • VS Code version: 1.60.2
  • Dart extension version: v3.27.0
  • Dart/Flutter SDK version:
  • Flutter (Channel stable, 2.5.1, on macOS 11.6 20G165 darwin-x64, locale en-CA)
    Dart SDK version: 2.14.2 (stable) (Wed Sep 15 12:32:06 2021 +0200) on "macos_x64
    not all run
    "
@lloydrichards
Copy link

Im getting the same issue when using the Dart: Run All Tests

What I've found is that you now have to use the VSCode test command Test: Run All Tests but this is now much much slower as it seems to be launching each time for each test file. For my TDD with over 60 test files this is now a really painful experience. Hopefully this is just a bug and the intended DX is still quick and responsive.

old Dart: Run All Tests -> 2-3sec
new Test: Run All Tests -> 40-60sec

@DanTup DanTup changed the title VSCode Test Runner doesn't run all tests "Test: Run All Tests" command does not run all tests Oct 6, 2021
@DanTup DanTup added this to the v3.28.0 milestone Oct 6, 2021
@DanTup DanTup added the in testing Relates to test execution of Dart/Flutter tests for end users label Oct 6, 2021
@DanTup
Copy link
Member

DanTup commented Oct 6, 2021

Thanks for the report! I overlooked the Test: Run All Tests command. I suspect it's running all discovered tests and therefore only runs those already visible in the tree. I think it may be fixed once #3591 is implemented, but I'll keep this open to track it to be sure.

The reason for Test: Run All Tests being slower might be that VS Code is handing us the full list of tests and they're being run one-at-a-time (or at least, one-file-at-a-time) whereas Dart: Run All Tests might be using a single test process (eg. just running dart test test). I've opened #3595 to look into that. Thanks!

@johnnypcc
Copy link
Author

johnnypcc commented Oct 6, 2021

Thanks for the report! I overlooked the Test: Run All Tests command. I suspect it's running all discovered tests and therefore only runs those already visible in the tree. I think it may be fixed once #3591 is implemented, but I'll keep this open to track it to be sure.

@DanTup You're welcome. But even after it discovers all tests, and I try to run them all, they're still not run.

@johnnypcc
Copy link
Author

this is now much much slower as it seems to be launching each time for each test file. For my TDD with over 60 test files this is now a really painful experience. Hopefully this is just a bug and the intended DX is still quick and responsive.

Yeah, it's much much slower :(

@DanTup
Copy link
Member

DanTup commented Oct 6, 2021

after it discovers all tests, and I try to run them all, they're still not run.

What's the relative path to one of your test files from the workspace folder? Eg. if you open a test file and find it in the Explorer tree, then right-click it and choose "Copy relative path", what does that look like? Is it outside of the test folder, or is the test folder not in the root of the folder opened in VS Code?

Can you reproduce it in a newly created project?

Yeah, it's much much slower :(

As a short-term workaround, you can still use the Dart: Run All Tests command for now which should behave the same as before the update (it should find all tests and be faster). The fix in #3595 will make the "Run Tests" button in the new VS Code runner the same speed as long as you're running all tests (eg. haven't hidden any) and are running without debugging. When debugging, or selectively picking tests (eg. excluding some), we need to run them by file, so that will be slower.

@johnnypcc
Copy link
Author

johnnypcc commented Oct 6, 2021

What's the relative path to one of your test files from the workspace folder?

All my tests, those which were run and those that weren't run, sit together under the test dir, which itself is in the root dir.
Thanks

@DanTup
Copy link
Member

DanTup commented Oct 7, 2021

@johnnypcc is it the same tests that are skipped each time? Are you able to make a small sample project that triggers the issue that you can share?

A log file might be useful too (note: it will include information about your tests, so best to capture in a project that's not sensitive):

  • Run the Dart: Capture Logs command from the command palette
  • Untick Analysis Server but leave the others ticked and click OK
  • Click the Run All Tests button, and wait for all sessions to complete (they're run sequentially)
  • Click Cancel on the logging notification to stop logging and open the log file

You can send the log to me at logs@dartcode.org (do not send anything sensitive), or you could try searching through it for some of the filenames that did not run, to see if there's anything obvious there.

@DanTup
Copy link
Member

DanTup commented Oct 7, 2021

I was able to reproduce this and figured out what it is. It happens when we run multiple test suites sequentially - the first one "ends" the test run and subsequent test results are not reported.

I have a fix for this, though I've also improved things so many test runs can run in a single debug session (which will make them much faster - #3599).

I may patch the first fix into a hotfix release, though the other one is tied in with some larger changes and will be in the next major version.

@DanTup DanTup closed this as completed in 035500e Oct 7, 2021
@DanTup DanTup modified the milestones: v3.28.0, v3.27.1 Oct 7, 2021
@DanTup
Copy link
Member

DanTup commented Oct 8, 2021

v3.27.1 has been published with the fix. Please upgrade (you can run the Extensions: Check for Updates command to force a check) and let me know if you're still having problems. Thanks!

@lloydrichards
Copy link

thanks so much for the hotfixes, has gone back to the same speed of testing using Dart: Run All Tests which is much appreciated. Looking forward to the next release and the other fixes 👍

@DanTup
Copy link
Member

DanTup commented Oct 12, 2021

Great, glad to hear! :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in testing Relates to test execution of Dart/Flutter tests for end users is bug
Projects
None yet
Development

No branches or pull requests

3 participants