Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dart: Run All Tests - producing strange test hierarchy and results #3603

Closed
moonytoes29 opened this issue Oct 11, 2021 · 4 comments
Closed
Labels
in testing Relates to test execution of Dart/Flutter tests for end users is bug
Milestone

Comments

@moonytoes29
Copy link

Describe the bug
In a multi package workspace consisting of an app and some libraries, when running the extensions Dart: Run All Tests the resulting hierarchy in the the testing panel is strange or wrong. Some tests look like they've been skipped, the tests under the file name are wrong or don't exist. Causing lots of issues identifying and fixing failing tests in a larger project. Only started happening after upgrading extension. Same weird output whether you are having Use VS Code test runner enabled or not.

This also introduces issues where you would right click on what you would expect to be a test under that file and click Go To Test only to have it point to the bottom of a file. Or tap Run test only to have the console spit out something like (No tests match regular expression "^adds one( (variant: .*))?$".)

To Reproduce
Steps to reproduce the behavior:

  1. Open the repro_workspace folder in vs code
  2. Execute Dart: Run All Tests

Expected behavior
The test files should be the top of the collapsable section with the tests within that file actually shown underneath with their success or failure status.

Screenshots

Screenshot 2021-10-11 at 10 08 24

Screenshot 2021-10-11 at 10 11 49

Versions (please complete the following information):

  • VS Code version: Version: 1.61.0
    Commit: ee8c7def80afc00dd6e593ef12f37756d8f504ea

  • Dart extension version: v3.27.1

  • Dart/Flutter SDK version: Flutter 2.2.3

Workspace to reproduce issues in:
repro_workspace.zip

@DanTup DanTup added the in testing Relates to test execution of Dart/Flutter tests for end users label Oct 11, 2021
@DanTup DanTup added this to the v3.27.2 milestone Oct 11, 2021
@DanTup
Copy link
Member

DanTup commented Oct 11, 2021

I was able to trim the repro down to just a simple folder with two tests and still see this:

Screenshot 2021-10-11 at 10 05 47

It looks like we assume we'll get a Suite event before each suite runs, and there will be no overlap, but I'm no longer certain that's a good assumption - though shouldn't be too complicated to remove/fix.

@moonytoes29
Copy link
Author

@DanTup - so if this has been closed. Do we have a patch release ETA?

@DanTup
Copy link
Member

DanTup commented Oct 11, 2021

@moonytoes29 still working on it. The fix is in master which is why this closed, but I'm still testing it cherry-picked into a patch of 3.27. I'm hopeful it won't be long though :-)

DanTup added a commit that referenced this issue Oct 11, 2021
The code previously assumed events would arrive in the order:

suite 1
test 1
testDone 1
suite 2
test 2
testDone 2

However the "suite 2" event could arrive before testDone for the tests in the previous suite. This changes to look up suites correctly, rather than assuming order.

Fixes #3603.
@DanTup
Copy link
Member

DanTup commented Oct 11, 2021

This should be fixed in v3.27.2. Please let me know if you still see issues (or confirm it's resolved). Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in testing Relates to test execution of Dart/Flutter tests for end users is bug
Projects
None yet
Development

No branches or pull requests

2 participants