Fine Code Coverage extension forcing all tests to run even when running a single test

[Pompeoar]

Installed product versions

• Visual Studio: 2019 Enterprise
• This extension: 1.1.11
• xunit 2.4.1
• xunit runner 2.4.3
• .dotnet core 3+

Description

With the extension enabled, trying to run a single test results in the test being run, and then all tests being run.

Steps to recreate

1. Create any new dotnet core project
2. add a dotnet core test library using xunit dotnetcore template
3. create a Test1 and Test2 fact
4. Set Test2 to Assert.IsTrue(false) to force a failure
5. Run only Test1

Current behavior

1 test passed ... then 1 Failed and 1 Passed.

Expected behavior

1 test passed.

Side Notes

This is particularly troublesome for larger test libraries and integration tests in particular.
I spent some time troubleshooting xunit and then visual studio settings before I found this. Disabling things like "discover tests in realtime" (options > test > general) does not fix this.

Pics to help: Note I did not run the second test

And yet both ran

[FortuneN]

I feared this curve ball would show up in the issue list some day.

Under the covers, the extension runs:

• Coverlet for .net core projects
  https://github.com/OpenCover/opencover/wiki/Usage
• OpenCover for .net framework projects
  https://github.com/opencover/opencover/blob/master/main/OpenCover.Documentation/Usage.pdf

Unfortunately, neither allows for fine-grained single (or a few) test execution/coverage scenarios.
We could ask the developers of those projects to add that feature but I'm not optimistic we would get that any time soon if at all.

I sincerely apologize for this. I don't think i can deliver this feature.
Still, please post the issues there; link back to this one and maybe we'll get a positive response (they are aware of this project; they may just throw us a bone)

[alex-jitbit]

@FortuneN When running vstest.console via opencover you can pass /TestName:TestMethod1 to vstests, via opencover -targetargs switch

[FortuneN]

Thanks @alex-jitbit

I figure i can also use module and class filters (which would be compatible for both openCover and coverlet down to single class level; i'm hoping to always be able to support the same features for both)

The missing part of this puzzle is "how do i know which tests were run".
My visual studio extension development knowledge is still low and my preliminary research has been unfruitful.
Can you help?

FYI: I'm reopening the issue with this renewed interest.

[tonyhallett]

@Pompeoar You could just disable the addin until you need to get coverage.

[Pompeoar]

@tonyhallett That's pretty much how I've worked around it.

[FortuneN]

@tonyhallett Maybe we could provide a simple Enable/Disable toggle on the Fine Code Coverage display window. This is far from the required solution but it could be a convince to be able to easily switch.

[tonyhallett]

@FortuneN I was going to suggest that. As I said to you I have an idea. For coverlet it would mean lo longer running coverage after tests have run. The collector driver facilitates this. Fine Code Coverage would need to just watch the TestResults folder for the emitted cobertura file. The difficulty is knowing this path as it is specified in .runsettings and is not available by reflection from test container. There is an exported internal class that provides this information for each test container.
The run settings are available to a test adapter implementing ITestDiscoverer and retrieved in discovery. Through refection the setter could be used to change these settings to our advantage.

Getting these run settings is necessary to replicate Vs regardless of the collector. This is not currently implemented.

An alternative is just a button and textbox for runsettings path and then the TestResults path and the Data collector can easily be added by Fine Code Coverage.

After Christmas I can provide further details

[tonyhallett]

In addition I think that when tests have finished it is possible to get stats on tests passed failed skipped through reflection.

If not using the data collector and having to re-run tests these values could be used to determine if coverage should be run.