Extension Test API - run results are not correctly updated when using multiple vscode.TestRun objects

[jimasp]

Does this issue occur when all extensions are disabled?: Yes (with the exception the extension code in question)

• VS Code Version: 1.77.2 (but this bug is not new)
• OS Version: Ubuntu 22.04

Imagine you have a multiroot workpace containing 3 project folders.
As an extension author, you want to create a vscode.TestRun object for each project folder, and kick off all three test runs asynchronously (i.e. in parallel) from a single vscode.TestRunRequest.

Unfortunately, this results in:

• run.passed not being honoured by the UI until the end of a different test run (or potentially all runs)
• run.skipped not being honoured at all (showing as not run rather than skipped).
  (run.failed works correctly)

Given the above, if the code sequence is:
run.enqueued(test) -> run.started(test) -> run.passed(test) -> run.skipped(test2) -> run.end()
Then what the user will actually see in the UI is as if the code sequence had been:
run.enqueued(test) -> run.started(test) -> run.enqueued(test) -> run.end() -> run.passed(test)
( run.skipped(test2) has no UI effect and the test is shown as not run)

i.e. the test items are stuck in a run.enqueued state until the end of a run (except for run.failed)
What is stranger is that it's not the run that is ended that has its tests updated, but a different run.
The run objects seem to be interacting with each other, i.e. they are not isolated.
This lack of isolation between run objects is confusing as an extension author (and most likely unintended, as it is not mentioned in documentation).
Equally, run.appendOutput also uses an output terminal shared between all run objects and therefore if all tests are run, then the output from all project folders (likely with different test runners) is intertwined and not isolated. This is a bit odd conceptually, given the api method is on the distinct TestRun object and not the shared RunRequest object.

The above issues could be solved by using a single run object for all project folders. However, the docs indicate that creating separate run objects is up to the extension author (without any caveats or warnings). At any rate, this approach would not help in the case of the next example...

Imagine there are two test extensions from different authors in the same workspace.
These extensions create their own vscode.TestRun objects, but what happens is:

• tests are not fully updated until the other extension ends its run
• run.appendOutput of the two run objects (and therefore two extensions) is intertwined in the test output terminal window (and testresults view), consequently any test output is not sequential or human-readable, such that the extension authors must use an extension-specific output window instead to isolate their extension test output (especially in the case of BDD output). Alternatively they could buffer the output, but buffering is not a good experience for the user. Guidance on the intended MS design to avoid intertwined test output between extensions would be appreciated, given separate run objects are not sufficient to isolate the output. Possibly the docs should be updated?

The extension author may reasonably, but wrongly, assume that (other than the sharing of vscode.TestRunRequest) the test runs are isolated and do not interact, and may not realise these issues exist during their testing unless they are running all tests and have another active test extension installed with its own tests.
Thus the manifestation of the results not being updated until the end of the run has previously been observed with two of Microsoft's own extensions and raised here.

Here is a gif of the behaviour using a simple (single) extension with three run objects, one per project folder.
We can see when the run result has been updated in the terminal window, but non-failed tests are stuck in a queued state in the UI until a different run or all runs are ended by clicking OK. Additionally we can see that skipped tests are never updated correctly, and that all runs use the same terminal window and so the output is intertwined between the runs.

Below is the code that produced the above gif.
Note that the code intentionally does not use a real mulitroot workspace, just a simulation of one. This is to show that the problem is not limited/caused by multiroot workspaces (or multiple extensions) but simply because run objects are not isolated from each other.
Another thing that is not shown in the gif that I noticed afterwards is that the "Test Results" window also looks wrong, only the first run (project 1) has expandable nodes.

// extension.ts

import * as vscode from 'vscode';

const projectFolders = ['project1', 'project2', 'project3'];

export function activate(context: vscode.ExtensionContext) {
	const ctrl = vscode.tests.createTestController('runtests', 'runtests');
	context.subscriptions.push(ctrl);
	const runHandler = testRunHandler(ctrl);

	// simulate multiroot workspace with 3 project folders
	for (const name of projectFolders) {
		const projectItem = ctrl.createTestItem(name, name);
		ctrl.items.add(projectItem);		
		createTestsForProjectFolder(ctrl, projectItem);
	}


	ctrl.createRunProfile('Run Tests', vscode.TestRunProfileKind.Run,
		async (request: vscode.TestRunRequest) => {
			await vscode.commands.executeCommand("testing.clearTestResults");			
			await runHandler(request);
		});
}

function createTestsForProjectFolder(ctrl: vscode.TestController, parent: vscode.TestItem) {
	const outcomes = ["pass", "fail", "skip"];
	for (const outcome of outcomes) {
		const id = `${parent.label} ${outcome}`;
		const test = ctrl.createTestItem(id, id);
		ctrl.items.add(test);
		parent.children.add(test);
	}
}

function testRunHandler(ctrl: vscode.TestController) {
	// kick off each project folder test run async (i.e. in parallel)
	return async (request: vscode.TestRunRequest) => {
		for (const name of projectFolders) {
			const run = ctrl.createTestRun(request, name);
			const projectQueue = getProjectQueue(ctrl, name);
			ctrl.items.forEach(test => {if(test.id.startsWith(name + " ")) projectQueue.push(test);});
			projectQueue.forEach(test => run.enqueued(test));
			runProjectQueueAsync(run, projectQueue);
		}		
	};
}

async function runProjectQueueAsync(run:vscode.TestRun, projectQueue:vscode.TestItem[]) {
	for(const item of projectQueue) {
		// run queue sequentially
		if(run.token.isCancellationRequested)
			break;
		run.started(item);			
		await new Promise(r => setTimeout(r, Math.random() * 5000));
		switch(item.label.split(" ")[1]) {
			case "pass": run.passed(item); break;
			case "fail": run.failed(item, new vscode.TestMessage("failed")); break;
			case "skip": run.skipped(item); break;
		}
		run.appendOutput(item.label + " result updated\r\n", undefined, item);
	}
	await vscode.window.showInformationMessage(`all tests updated for ${run.name}. click OK to end run ${run.name}`, "OK");	
	run.end();
}

function getProjectQueue(ctrl:vscode.TestController, projectFolerName:string) {
	const pqItem = ctrl.items.get(projectFolerName);
	const queue: vscode.TestItem[] = [];
	pqItem?.children.forEach(c => queue.push(c));
	return queue;
}

Full repo for the above example is here, but note that the repo is basically a duplicate of the MS "test-provider-sample" except that extension.ts is replaced with the above code, so alternatively you can just copy/paste the above code into that file if you already have the MS repo locally.

[connor4312]

I really appreciated the detailed issue and reproduction here! I think there was an issue that was probably the same root cause from a while ago, but the reporter never provided a repro and I don't think I was doing the right steps that you've laid out here. (And I can't seem to find it right now.)

I'll get it fixed.

[VSCodeTriageBot]

This bug has been fixed in the latest release of VS Code Insiders!

@jimasp, you can help us out by commenting /verified if things are now working as expected.

If things still don't seem right, please ensure you're on version d6290ef of Insiders (today's or later - you can use Help: About in the command palette to check), and leave a comment letting us know what isn't working as expected.

Happy Coding!

[lramos15]

@connor4312 Any verification steps here? I copied the example code into the sample test provider as suggested and then ran the sample. I see this which doesn't seem right.

[connor4312]

Ah, yep, one more spot that needed nuking

[jimasp]

@connor4312 @lramos15

I've tried these changes (1.78.0-insider).

Are all the changes in 1.78.0-insider?

The good:

• run.skipped no longer shows as not run
• test run output is now separated in the terminal

The bad:

I still get the UI behaviour shown in the above gif, for example:

project 2 updates all tests -> project 2 test UI is not updated -> project 3 run is ended -> project 2 test UI is updated

i.e. I still don't understand why I have to run.end(project3) to get project2 to update its test results?

In an attempt to workaround this issue, I've tried using the modified code below.
This uses:

• a separate TestController for each workspace folder
• a separate runHandler for each workspace folder
• a separate TestRunProfile for each workspace folder
• a separate TestRun for each workspace folder (unchanged from previous code)

But this doesn't help. The behaviour is unchanged.
Tests are still stuck in a queued state until a different test run is ended, or until all test runs are ended.

What can I do to stop these test runs from blocking each other's test results?

Sidenote: this modified code is presumably the same in terms of vscode instance object isolation as would be the case with a separate extension acting on each folder, i.e. separate extensions can currently block each other's test run results.

// extension.ts

import * as vscode from 'vscode';

export function activate(context: vscode.ExtensionContext) {
  // simulate multiroot workspace with 3 project folders
  for (const name of ['project1', 'project2', 'project3']) {
    const ctrl = vscode.tests.createTestController(name, name);
    context.subscriptions.push(ctrl);
    const runHandler = testRunHandler(ctrl, name);
    createTestsForProjectFolder(ctrl);

    ctrl.createRunProfile(`Run Tests:${name}`, vscode.TestRunProfileKind.Run,
      async (request: vscode.TestRunRequest) => {
        await UI.clearTestResults();
        await runHandler(request);
      });
  }
}

function createTestsForProjectFolder(ctrl: vscode.TestController) {
  for (const outcome of ["pass", "fail", "skip"]) {
    ctrl.items.add(ctrl.createTestItem(outcome, outcome));
  }
}

function testRunHandler(ctrl: vscode.TestController, name: string) {
  return async (request: vscode.TestRunRequest) =>
    runProjectQueueAsync(ctrl.createTestRun(request, name), ctrl);
}

async function runProjectQueueAsync(run: vscode.TestRun, ctrl: vscode.TestController) {
  const projectQueue: vscode.TestItem[] = [];
  ctrl.items.forEach(c => projectQueue.push(c));

  for (const item of projectQueue) {
    // run queue sequentially
    if (run.token.isCancellationRequested)
      break;
    run.started(item);
    await new Promise(r => setTimeout(r, Math.random() * 5000));
    switch (item.label) {
      case "pass": run.passed(item); break;
      case "fail": run.failed(item, new vscode.TestMessage("failed")); break;
      case "skip": run.skipped(item); break;
    }
    run.appendOutput(item.label + " result updated\r\n", undefined, item);
  }

  await vscode.window.showInformationMessage(`\nAll tests have been updated for ${run.name}\n\nClick OK to call run.end() on ${run.name}'s TestRun`, { modal: true });
  run.end();
}


class UI {
  private static clearing = false;
  private constructor() {/**/ }
  static async clearTestResults() {
    // clear test UI results ONCE per TestRunRequest  (this is important when testing this bug, because otherwise you could be looking at the UI results of the previous run)
    if (!UI.clearing) {
      UI.clearing = true;
      await vscode.commands.executeCommand("testing.clearTestResults");
      UI.clearing = false;
    }
    while (UI.clearing) {
      await new Promise(r => setTimeout(r, 10));
    }
  }
}

updated gif with the above code running in 1.78.0-insider

in this gif, we can see that:

• project1 does not update until project2's run is ended
• project3 does not update until project1's run is ended

[connor4312]

Apologies, I missed the followup comment. Insiders builds happen during CET mornings Mon-Fri, and since this got merged on Friday the next build wasn't until the following Monday, which was after you commented. I've tried your updated code sample and it looks like the issue is fixed. Please let us know if you hit any other snags, and once again thank you for the very nice bug reports and repro steps!

[joyceerhl]

Looks good now with the code from #180041 (comment)