[Testing] Coverage API Issues: Monorepo & Accumulation

[connectdotz]

Type: Bug

As we integrate our extension, vscode-jest, with the newly available Testing Coverage API, we've identified some issues that notably impact usability. Below, I provide some video demonstrations from our development environment, followed by descriptions of each issue.

Issue 1: Incomplete Coverage Trigger in Monorepo Projects

In monorepo projects with multiple testControllers, initiating a top-level coverage run does not trigger all registered controllers. The coverage run is activated only for the first controller.

Screen.Recording.2024-04-16.at.5.58.05.PM.mov

video description:
In our setup with 'jest snapshot' and 'jest ts' controllers, initiating coverage from the top-level menu only triggered the "jest snapshot" controller's runHandler.

Issue 2: Non-Accumulative Coverage Display

The test coverage display currently does not accumulate data across test runs. This is a concern because generating coverage information usually requires significant resources, and users could execute partial coverage runs. Without cumulative coverage, historical data is not preserved, which can reduce the feature's usability. Ideally, test coverage could mirror the behavior of test run status, which consistently updates and maintains the latest information, regardless of the test run sequence.

Screen.Recording.2024-04-16.at.6.14.36.PM.mov

video description:

1. Run coverage for "jest snapshot" => coverage info correctly displayed as expected.
2. Run coverage for "jest ts" => The "jest snapshot" coverage was replaced by the "jest ts" coverage in the panel.
   desired: both coverage info should be visible in the panel.
3. Run a regular test in "jest snapshot" => The coverage panel disappeared.
   desired: The coverage panel should still be visible.
4. Run "Test: Open Coverage" command didn't bring back the latest coverage panel either.
   desired: The command should bring back the latest coverage panel

VS Code version: Code 1.88.1 (e170252, 2024-04-10T17:42:52.765Z)
OS version: Darwin x64 23.4.0
Modes:

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-8700B CPU @ 3.20GHz (12 x 3200)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	2, 2, 2
Memory (System)	32.00GB (1.90GB free)
Process Argv
Screen Reader	no
VM	0%

Extensions (48)

Extension	Author (truncated)	Version
better-comments	aar	3.0.2
vscode-color-pick	ada	0.6.2
emojisense	bie	0.10.0
vscode-markdownlint	Dav	0.54.0
vscode-eslint	dba	2.4.4
xml	Dot	2.5.1
gitlens	eam	14.9.0
prettier-vscode	esb	10.4.0
vscode-commands	fab	2.0.2
flow-for-vscode	flo	2.2.1
shell-format	fox	7.2.5
html-preview-vscode	geo	0.2.5
copilot	Git	1.180.821
copilot-chat	Git	0.14.1
vscode-pull-request-github	Git	0.86.1
go	gol	0.41.2
svg	joc	1.5.3
vscode-insertdatestring	jsy	2.3.1
vscode-colorize	kam	0.11.1
json-lines-viewer	leh	0.0.4
git-graph	mhu	1.30.0
vscode-docker	ms-	1.29.0
debugpy	ms-	2024.4.0
isort	ms-	2023.10.1
python	ms-	2024.4.1
vscode-pylance	ms-	2024.4.1
jupyter	ms-	2024.3.1
jupyter-keymap	ms-	1.1.2
jupyter-renderers	ms-	1.0.17
vscode-jupyter-cell-tags	ms-	0.1.9
vscode-jupyter-slideshow	ms-	0.1.6
remote-containers	ms-	0.354.0
wordcount	ms-	0.1.0
vscode-jest	Ort	6.3.0
material-icon-theme	PKi	4.34.0
java	red	1.29.0
markdown-preview-enhanced	shd	0.8.13
code-spell-checker	str	3.0.1
pdf	tom	1.2.2
intellicode-api-usage-examples	Vis	0.2.8
vscodeintellicode	Vis	1.3.1
vscode-java-debug	vsc	0.57.0
vscode-java-dependency	vsc	0.23.6
vscode-java-pack	vsc	0.26.0
vscode-java-test	vsc	0.41.0
vscode-maven	vsc	0.44.0
vim	vsc	1.27.2
markdown-all-in-one	yzh	3.6.2

(1 theme extensions excluded)

[connor4312]

Non-Accumulative Coverage Display

I'm not sure this is safe to do. For test items we can do that since the state of a test item is inherently tied to its existence in the tree. That's not true for coverage: I could have a test run that generates coverage for a certain file, and if I move/rename/delete that file, the old coverage for the file would be "stuck" unless I cleared all previous coverage data. This could also confusingly influence coverage statistics if a user runs their entire test suite with coverage and their coverage numbers are influenced by data for files that no longer exist.

[connectdotz]

@connor4312, thank you for your insights on this matter. Here are a few thoughts in response:

I could have a test run that generates coverage for a certain file, and if I move/rename/delete that file, the old coverage for the file would be "stuck" unless I cleared all previous coverage data.

Considering that each FileCoverage corresponds to a source file, it seems feasible for the provider—or perhaps even the Test Explorer itself—to manage these through life-cycle events. For instance, coverage could be automatically removed or reset when files are moved, renamed, or deleted, assuming there's an API to communicate such changes back to VSCode.

This could also confusingly influence coverage statistics if a user runs their entire test suite with coverage and their coverage numbers are influenced by data for files that no longer exist.

Allowing providers to opt-in to maintaining FileCoverage integrity as described above could mitigate this concern. Additionally, it might be beneficial to implement a "reset" function or possibly use a collection like TestItemCollection to manage the coverage tree comprehensively.

At a minimum, I believe the coverage panel should maintain separate coverage information for each controller. Initiating a coverage run in one controller should not result in the loss of coverage data from another controller.

Thank you once again for considering these suggestions. I look forward to your thoughts on these potential improvements.