API for async test items

[jrieken]

This is part of #122208 and a continuation of #115089.

Today, there is two APIs for asynchronously resolving test items:

• the TestItem.status: TestItemStatus-API which is what @hbenl has suggested in Allow tests to be loaded asynchronously #115089 (comment). It allows extension to signal that an item isn't fully loaded yet.
• the TestController.resolveChildrenHandler which tells extensions that an item should become "Resolved" and when an item isn't interesting anymore (via the cancellation token)

There is the following issues with this approach

• such use of the cancellation token in the resolveChildrenHandler is a first for the API and each "first" is being questioned by default
• both APIs are coupled. An item cannot be "Pending" when its controller doesn't support to resolve children. I see how this maybe helps with correctness but it seems confusing. An extension should have the freedom to create unresolved items and resolve them at their own schedule and the UI (VS Code) should reflect that.

---

I also want to take a steps back and revalidate that we still need async test items. Many changes have happened, the TestController now sports a push-model and that allows extensions to add and remove test items any time and by their own schedule. The overall set of test items is fluid, there is no more thing like onDidDiscoverInitialTests, and extensions are in full control.

So, the only use-case of async test items is when extensions "know that a file/thing contains tests but hasn't looked into it". The only value of that is the rendering of an empty tree node in the testing explorer.

⬆️ @hbenl, @jdneo, @matepek, @OmarTawfik How often is this a problem? Assuming you can add/remove resolved test anytime and assuming you can lets us know that you are currently computing tests would still need to support for unresolved, shallow test items that need to be filled in later?

[jdneo]

@jrieken One benefit of async loading test items is that in the test explorer, the tests can be loaded level by level(when user expands the node deeper and deeper). Especially in a large project which might contains thousands of test items, the extension doesn't have to load all the tests at one time.

So one (maybe silly) question: If we make the test items synchronized, will the extension get a signal when a test node is expanded? (like currently, TestController.resolveChildrenHandler will be called)

[matepek]

Language: C++

Happens that a user complains about the slow loading of the tests. Async would solve this.

What I think is best for my case:

• Loading the executable list as resolvable test items.
• Loading the executables one by one in some order and resolve them when they finishes.
• If the user expands an item meanwhile the adapter(me) reprioritize the loading to favor the one the user curious about.
• User is happy :)

[jrieken]

Understood - so this is mostly about breath first instead of depth first. It makes sense to me but it also means we need to polish the resolveChildrenHandler further.

The catch with resolveChildrenHandler is that is has a misleading name. We usually use "resolveXYZ" in providers and there it means "provide more data before VS Code proceeds", like resolveCodeLens. For test controller is means the following:

• Items that aren't pending are ignored (we shouldn't do that)
• Items must be resolved via the handle but they don't signal "resolved" via a returned promise but via state-updates.
• Items must be watched/updated until the passed cancellation token flips

That means the "resolving" applies to a much longer span than the execution of the handler and likewise the cancellation token represents the period for which VS Code has "special" interest in an item.

The alternative proposal would do away with the resolve-callback and add an event which fires whenever a test item is interesting (expanded in the tree, opened as text document) and vice versa. Something like

interface TestController {
  readonly onDidChangeInterest: Event<{ item: TestItem<T>, interesting: boolean }> // needs better names
}

VS Code would "blindly" fire these events and extensions can use this as signal to prioritize resolving of items, and/or start watching for changes etc. Similar to how extensions are listening to other events, like onDidChangeTextDocument or onDidSaveTextDocument.

[connor4312]

Joh and I talked a bit about this this morning. What we're thinking of is making three changes:

• Keep the resolveChildrenHandler, and remove the CancellationToken parameter
• Remove TestITem.status and instead have TestItem.canExpand?: boolean
• Add TestItem.busy?: boolean. Note that the UI would implicitly show the item as busy while resolve is being called.

This solves a few problems:

• The 'weird' use of cancellation token in the resolve handler
• It removes the implicit "subscribing" functionality of the handler and, and lets the extension do what it needs. This dovetails in with what the current set of test providers do, where they create watchers and pretty much manage the lifecycles for themselves.
• It decouples business from expand state, allowing a good way to show business outside the initial expand request (e.g. on refresh)

[connor4312]

I've made these changes in df5cd47. However I went with canResolveChildren instead of canExpand. canExpand is not a very good name: the editor wants to know if it should go to the resolveHandler to ask for more children. This is separate from whether an item is expandable or not. For example, an item that represents a test in a file might have children and be expandable, but receive tests only from its parent and not 'resolvable' by itself.

[jrieken]

@hbenl, @jdneo, @matepek, @OmarTawfik Sorry for another ping but I wonder "how deeply asynchronous" your tests are. I understand that discovery and processing of test files is two steps but I wonder how many async steps follow when a test file is processed:

1. discover test files/containers like src/vs/base/test/common/arrays.test.ts
2. per file/container discover tests (suites, unit tests, etc)

I wonder, iff step 2 is ever async/lazy or always a single sync pass. E.g. we delay loading a Java class file but once it is loaded discovery of tests is a single pass and sync, right? The same for AST- or execution-based discovery, right? Iff so we might be able to simplify the API further, e.g have managed test containers and short lived test items. The advantage would be that extensions only need to create and return test items from their source of truth but never dispose nor update them (VS Code would handle all of that). The disadvantage is that some flexibility/complexity goes, esp. the case of "i have lazily discovered this suite and I now delay discovery of its tests because ???" Now I wonder if multi-level lazy test items are real or just a theoretical problem?

There is some API sketch for how it could look here: #127143 (comment)

[jdneo]

Some input from Java:

Test Hierarchy

In Java, the test hierarchy looks like this:

Root
├── Java Projects
    ├── Package Fragments (folders)
        ├── Types (files)
            ├── Methods (unit tests)

Test Discovery -- All async calls

1. Find all Java projects in the workspace
2. Find all testable types in a Java projects and their belonging package fragments
3. Find all testable methods in a testable type

About the parent pointer

Currently, the parent pointer can be useful in one scenario:

In Java, we allow other extension request a customized test running and send that request to the test runner extension. The workflow looks like this:

Other extension register their commands to the test explorer via testing/item/context. Then they can get the test node, and resolve their own launch configuration based on the selected node. Then they call a command from the test runner extension, passing the selected node and the launch configuration, the test runner runs it and show the result.

The problem here is, since the test node passed to the test runner is a different instance than the real one that is created by the test runner itself. So currently we use the parent pointer to trace back from the target node to root in the test tree, and then use that path to find the real node.

This is because we use a WeakMap to store the data for each node, so the instance must be the same to get the data information.

Though we can use Map and the id as the key to get rid of that, the advantage of using WeakMap is that we don't need to take care about deleting entries as long as VS Code will dispose the test item.

[jrieken]

Test Discovery -- All async calls
Find all Java projects in the workspace
Find all testable types in a Java projects and their belonging package fragments
Find all testable methods in a testable type

@jdneo What is corresponding output in terms of vscode.TestItem instances? Do you have a test item per project with children for each testable type with children for each method? Or is it testable type at the root?

Do you have a code pointer for me so that I can understand how the current API is being used?

[jdneo]

Yes, all the node belongs to the test hierarchy will be a test item. The layout looks like this:

Meanwhile, you can find the build here if you want to have a try: microsoft/vscode-java-test#1162.

The implementation can be found here: https://github.com/jdneo/vscode-java-test/blob/cs/new-test-api/src/controller/testController.ts

[jrieken]

Very helpful. Thanks @jdneo.