Testing API: "Retired" TestItem "state"

[matepek]

There was a feature request: #115086

In case of the final API I don't know how to implement it. I cannot find the feature.
I guess it is removed. I think my users would love the feature.

Would you help me on this @connor4312 ?

[matepek]

Also there is a feature related to this which might worth to be considered. Test Explorer had an "auto-run" option.
If it is checked and there is a retired event about a test then a test-run will be created for those. (in case of test-explorer)

I can manually add option like this but it is a nice feature I believe.
A "Run retired/stale tests only" command can be useful also... maybe...

[connor4312]

The Test Explorer UI extension used retired states to drive autorun. I would instead like to have the extension drive its auto autorun (#134941), so retired states are no longer necessary for this use case and I'm not sure whether I want to bring them back.

[matepek]

The retired state basically an indicator in the tree view and the user can see if a test's result is outdated.
In case of c++ test executables I can detect the change of the moditime and mark all the tests coming from the executable as outdated. I love this feature. The way ho it does it that the retired tests' icons are 20% transparent I think. (Tooltip also could indicate it) Or maybe if I could just clear the state but that is less useful because it is nice to know what was the previous state of the test.

The auto-run is built upon this indeed. I can implement it easily if you don't wish to add it.
Regarding to this: It would be nice if we could contribute to the "..." menu. (adding new items. In this case: one with a checkbox)

[connor4312]

You can contribute to all test menus like any other menu 🙂 https://code.visualstudio.com/api/references/contribution-points#contributes.menus

[matepek]

Hello,
Any news on this front?

[matepek]

Hello,
Any news on this front?

[matepek]

Hello,
Any news on this front?

[xisui-MSFT]

Hi @connor4312, is there a way to mark some tests as outdated without auto rerun? In our extension, we would like to mark some tests as outdated in cases like some configs change, but we prefer not to do auto rerun since rebuilding the project may take a long time.

[connor4312]

I've implemented proposed API for this:

declare module 'vscode' {

	export interface TestController {

		/**
		 * Marks an item's results as being outdated. This is commonly called when
		 * code or configuration changes and previous results should no longer
		 * be considered relevant. The same logic used to mark results as outdated
		 * may be used to drive {@link TestRunRequest.continuous continuous test runs}.
		 *
		 * If an item is passed to this method, test results for the item and all of
		 * its children will be marked as outdated. If no item is passed, then all
		 * test owned by the TestController will be marked as outdated.
		 *
		 * Any test runs started before the moment this method is called, including
		 * runs which may still be ongoing, will be marked as outdated and deprioritized
		 * in the editor's UI.
		 *
		 * @param item Item to mark as outdated. If undefined, all the controller's items are marked outdated.
		 */
		invalidateTestResults(item?: TestItem): void;
	}
}

I've gone back and forth on the method name a few times 🤷 Please let me know if there are any concerns with this. You can try it out in the next Insiders.

[matepek]

Hello,
Two things.

1. For TestMate C++ in 99% of the cases it will have a set of items instead of just one item. API could be friendlier but I can use a for loop too.

2. I'm not sure how will it work together with continuous run. As you said the "The same logic used to mark results as outdated may be used to drive continuous test runs." I assume it will be order sensitive so I should mark the tests outdated then trigger the continuous run otherwise I might invalidate fresh results. Is my assumption correct?

[connor4312]

For TestMate C++ in 99% of the cases it will have a set of items instead of just one item. API could be friendlier but I can use a for loop too.

Sure, I will have it take both an array as well as a single value.

I assume it will be order sensitive so I should mark the tests outdated then trigger the continuous run otherwise I might invalidate fresh results. Is my assumption correct?

Correct, if you have ideas to clarify wording let me know:

   * Any test runs started before the moment this method is called, including
   * runs which may still be ongoing, will be marked as outdated and deprioritized
   * in the editor's UI.

[connor4312]

Finalized in #188180

[alexr00]

@connor4312 what is the best way to verify this?

[connor4312]

1. Use the test provider sample https://github.com/microsoft/vscode-extension-samples/tree/main/test-provider-sample
2. Call ctrl.invalidateTestResults(file) here https://github.com/microsoft/vscode-extension-samples/blob/main/test-provider-sample/src/extension.ts#L134
3. F5, run all tests, and change a file. Validate that the node in the test explorer gets dimmed and no "proposed" errors are thrown when calling the API.

[rzhao271]

Verified, though it was easier for me to see the change with a dark theme than a light theme.