Multi package workspace - can't find all tests to execute

[moonytoes29]

Describe the bug
A bug in the test finder and therefore the overall number of tests ran.

When in a multi package workspace, if the package folders are all at the same directory level, then the test finder discovers all of the tests and runs them. See packages_all_on_one_level attached zip. This finds all of the workspace's 702 tests and runs them correctly.

When the packages are at another folder level deep. The test finder doesn't find all the tests. Having a main app folder and a packages, shared or features folder for other packages is a common workspace structure for mono repo applications. So having this issue is a blocker for this extension.

When you open the folder for packages_one_level_deep attached zip, you run and see that it only executes 312 tests. It misses tests that in subfolders of the shared packages.

I have also noticed an error, that if you run Test: run all tests before tapping on the test tab, it fails with an error

Final issue I'd like to raise is the naming of the test suites as it finds tests within a package. The test suites appear to be named after the first test that the test finder finds in the package, rather than any meaningful package scoped name. See image below.

To Reproduce
Steps to reproduce the behavior:

1. Open packages_all_on_one_level and go to test navigator and run all tests
2. See that there are the correct 702 Tests
3. Open packages_one_level_deep and go to test navigator and run all tests
4. See that there are the incorrect 312 Tests
5. Look in found tests and can see that it is missing tests located under subfolder1, anotherSubfolder3 and other subfolders containing 26 tests each.

Expected behavior
Finds all the tests in any folder, executes all and gives correct number of passed/failed tests.

Also allows for folders to be skipped would be nice to. As sometimes we have code in our workspaces that isn't ours :)

Screenshots
See screenshots above

Reproduced with:
packages_all_on_one_level.zip

packages_one_level_deep.zip

Versions (please complete the following information):

• VS Code version: 1.62.3
• Dart extension version: v3.29.0
• Dart/Flutter SDK version: Flutter 2.2.3

[moonytoes29]

As another note. The packages_one_level_deep issue is not present in older extension version v3.27.2.
When you use the now missing Dart: run all tests it finds and executes to full 702 tests.

Issue appears only with the auto discovery and then execution of tests.

[DanTup]

This was caused by a limit of walking only 5 levels when searching for tests (to avoid searching huge trees if someone opens their home directory or similar). This wasn't really the best way though, we already have a lot of code that handles locating projects (up to 3-4 levels deep), and once we've found a project there's no reason not to just walk the whole tree from there.

I've changed this to do that and raised the limit to 100 levels (a guard against something going wrong or walking cyclic symlinks or similar). In your test project, all tests (including those in the sub folders) now show up.

I've raised #3713 and #3714 for the other issues noted here.

[DanTup]

Fixes for the issues noted above have been included in today's pre-release build if you'd like to try them out before they make a stable release:

[moonytoes29]

@DanTup - are they ever going to make a stable release?

[DanTup]

Yep, they will be in the upcoming stable release. Releases are approximately monthly and there hasn't been one since these fixes were made. I expect to do one soon (if not this week, next).

There isn't much difference between the upcoming stable release and what's currently in the pre-release version though, so if you need the fix I'd recommend just using the pre-release channel until then. There are now new known issues in the pre-release (and if there are, they need identifying and fixing, else there'll end up in the stable release too).

[moonytoes29]

I'm on v32.0 release, I can't reproduce in workspace above. But in my work project, I'm still having issues. Where large number of tests aren't getting feedback in the test window. 559/559 tests passed. When we have around 2000 tests. Also have failing ones in the Debug Console, but nothing showing in the failed tag in the test window.

I'm really trying to reproduce in the above workspace. But for some reason that one is working okay.

[DanTup]

Are the tests missing from the test explorer tree, or just reported with the wrong status there? If they're appearing in the tree and the status or counts are bad, that may be a different issue (and worth filing a new issue for).

It might be useful to capture a log with Dart: Capture Logs (untick analysis server, but leave others ticked), then run all of the tests, then click Cancel on the logging notification to stop logging and open the log. You can send it to logs@dartcode.org if you don't want to share here (though don't send anything sensitive).

[moonytoes29]

They weren't missing, but the count wasn't working. Failed tests were not showing up and even if they were, fixing them and running again was not updating the status in the test section. When I cleared test results and ran all tests again, there was no feedback at all.

I've weirdly seemed to have fixed or massively improved the issue by turning off and on the Dart: Show Test Code Lens setting. Turning that off and running it, picked up count of all my tests and the correct failed tests were shown. Turning it on again and running seemed to have a similar result. Except that when I fixed the test and ran again, it wasnt updating the status to passed in the test window. However having that setting turned off, fixed that issue. And when I fixed it, it moved to passed. However, that setting is very useful for running individual tests in vscode.

So I don't have a reproducible example. But the fact that turning off, on and then ultimately leaving off the Dart: Show Test Code Lens has an affect, might lead you to an area to investigate.

For some context, my work project has around 1700 tests across multiple packages, tests can be multiple folders deep (fixed by last update).

Edit:

I don't know if it might help or find a cause. But we have an addition to launch.json settings, for a code lens for golden tests.

  {
            "name": "Golden",
            "request": "launch",
            "type": "dart",
            "codeLens": {
              "for": ["run-test", "run-test-file"]
            },
            "args": ["--update-goldens"]
        },

I can try adding that to a repro workspace to see if we get issues.

[DanTup]

I can't think of any reason the code lens setting would affect this, could it be coincidence?

If you can get steps for me to repro in a sample project though, it'd be good to figure out. The counts only show the tests that most recently ran in a session. Previously there was a bug where some types of invocations of tests would run them in multiple sessions, but without attributing them to the same "test run", so I wonder if it's something similar.

Although, I can't explain why filtering to @failed wouldn't work if they're in the tree, that's entirely VS Code behaviour. Were the nodes marked as failed or not in the tree?

[moonytoes29]

I've just turned it back on, cleared the results and ran my tests again. Now no feedback is showing in the test window.

Thats the thing, it wasnt showing feedback for ALL of my tests. So even though the console runs them all and gets feedback for failing ones. The panel wasnt setting that test to failed. Even if I actually right clicked on the test in the panel and tapped to run test. Still no feedback.

Turning off show test code lens. Clears the results and now when I run, its counting up and showing updates as it executes. Although actually, the counting has stopped at 700 now. Even though tests are continuing.

This is really weird behaviour tbh. Its becoming very frustrating.

Edit:

Closed VS Code and opened again and ran. This time even with setting off, it freezes count and feedback at 403.

I'm at a loss.... I've tried cleaning and reinstalling vs code tbh. Still no improvement.

[DanTup]

@moonytoes29 are you able to reproduce in a project you can share?

It may also be useful to capture logs with the Dart: Capture Logs command (untick Analysis Server but leave everything else ticked), and also check the VS Code console (Help -> Toggle Developer Console) to see if there are any errors listed.

[moonytoes29]

@DanTup - in my work project (which I can't share for obvious reasons). I ran with the Developer Console on and got some errors.

I've anonymised a bit and included the console logs.

The main issue is this error: mainThreadExtensionService.ts:63 [[object Object]]Attempted to insert a duplicate test item ID

When I navigate to the tests highlighted. I can't see what would cause the issue.

The first test culprit in the console is ab_pinned_bottom_layout_test.dart. That file uses the golden_toolkit package and has two golden tests in it. Both with unique names.

testGoldens('no pinned bottom', (tester) async {

testGoldens('pinned bottom', (tester) async {

If I close that file, clear dev console and reopen it ab_pinned_bottom_layout_test.dart, I get the attempted to insert a duplicated test item ID error. So its part of the code lens as it analyses the file.

I can try removing the golden tests and seeing if they're still an issue.

How is the test item ID calculated for the test runner?
devConsole.zip

I'll try see if I can reproduce with non work based project still. Will add golden_toolkit and some tests to see if that's causing some issues with calculating test ID.

[DanTup]

Looking at the call stack in that error, it might be a group rather than test triggering that message:

at TestModel.groupDiscovered (vscode-file://vscode-app/Users/my.user/.vscode/extensions/dart-code.dart-code-3.32.0/out/dist/extension.js:24116)

Although the code that rebuilds the group will also replace its children:

Dart-Code/src/extension/test/vs_test_controller.ts

Line 142 in c32340e

private createOrUpdateNode(node: TreeNode): vs.TestItem | undefined {

The IDs are made using the path and name:

Dart-Code/src/extension/test/vs_test_controller.ts

Lines 201 to 205 in c32340e

	private idForNode(node: TreeNode) {
	return node instanceof GroupNode || node instanceof TestNode
	? `${node.suiteData.path}:${node.name}`
	: node.suiteData.path;
	}

I'll do some more testing to see if I can repro, but please let me know if you can trigger it in a project you can share. Could you also file a new issue specific to this remaining problem? Thanks!

[DanTup]

I've managed to reproduce this. Not sure if it's the same thing you're doing, but I suspect the underlying bug is the same. I've filed #3776.

[DanTup]

@moonytoes29 the issues are fixed by 0464e3f (which prevents the error, but will show you some dupes in the tree) and eBay/flutter_glove_box#140 which will resolve the duplicates by correctly marking testGoldens as a function producing a test, not a group.

I'll push a build to the pre-release channel shortly if you'd like to get the crash fix without waiting for a stable release.

[DanTup]

@moonytoes29 if you want to test the fix:

(And although not necessary for this fix, I'd recommend the same on the Flutter extension to keep them in-sync, and switch both back to Stable when you decide to do that).