Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for dotnet test --collect:"Code Coverage" #981

Open
harshjain2 opened this issue Aug 4, 2017 · 194 comments
Open

Add support for dotnet test --collect:"Code Coverage" #981

harshjain2 opened this issue Aug 4, 2017 · 194 comments

Comments

@harshjain2
Copy link
Contributor

@harshjain2 harshjain2 commented Aug 4, 2017

Description

Add support for code coverage through dotnet test on Windows OS machine.
dotnet test --collect:"Code Coverage"

@harshjain2 harshjain2 added this to the 15.5 milestone Aug 4, 2017
@harshjain2 harshjain2 changed the title Add support for dotnet test --collector:"Code Coverage" Add support for dotnet test --collect:"Code Coverage" Aug 4, 2017
@arpit-nagar
Copy link

@arpit-nagar arpit-nagar commented Aug 4, 2017

Do you have any plans for Linux ??

(edit from @codito: there are atleast 22 more votes in #579 (comment) for code coverage support in linux. Using this comment to keep track of linux support request)

@nulltoken
Copy link

@nulltoken nulltoken commented Aug 4, 2017

Why the "having Visual Studio 2017 Enterprise installed" limitation? Is there any plan to lift it?

I believe collecting test code coverage during a CI build (with only the .NET Core SDK being deployed) would be a very valid scenario.

@ctro
Copy link

@ctro ctro commented Aug 7, 2017

👍 I'd very much love to have working code coverage within the confines of .NET Core.

@harshjain2
Copy link
Contributor Author

@harshjain2 harshjain2 commented Aug 9, 2017

@arpit-nagar test code coverage is VS enterprise and windows only so far, current plan is to get the same level of support in dotnet test (windows + VS enterprise). Mac/linux support isn't in our immediate backlog. /cc @pvlakshm @sbaid to consider the feedback for prioritization.

@arpit-nagar
Copy link

@arpit-nagar arpit-nagar commented Aug 9, 2017

@harshjain2 This is a very basic requirement. And as dotnet core is multi platform this feature should also need to be available on Linux.
Is there any way we can achieve code coverage on Linux for dotnet core project.

@andyjansson
Copy link

@andyjansson andyjansson commented Aug 9, 2017

test code coverage is VS enterprise and windows only so far, current plan is to get the same level of support in dotnet test (windows + VS enterprise).

@harshjain2 What's the rationale behind requiring VS enterprise for code coverage?

@rol-dave-overall
Copy link

@rol-dave-overall rol-dave-overall commented Aug 15, 2017

Downloaded VS2017 15.3 and .NET Core 2.0 and after entering (dotnet test --collect:"Code Coverage") I get an error:

The test source file "C:\Application2\tests\Console.UnitTests1\Coverage" provided was not found.

Tried changing this to be (dotnet test --collect:"CodeCoverage") i.e. without the space and then get this error:

dotnet exec needs a managed .dll or .exe extension. The application specified was 'C:\Program'
Object reference not set to an instance of an object.

@jstallm
Copy link

@jstallm jstallm commented Aug 18, 2017

It sounds to me like the status is that we cannot do code coverage for asp.net core 1.X using dotnet test at the command line. Is this correct?

@edumserrano
Copy link

@edumserrano edumserrano commented Aug 21, 2017

I get the same as @rol-dave-overall. This is a show-stopper for us trying to get our solution build on Visual Studio Team Services. Disabling code coverage is not really an option.

Any available workarounds?

@RoCore
Copy link

@RoCore RoCore commented Aug 23, 2017

actually have the same problem. the vso test with --collect Coverage or "Code Coverage" not working. the runsettings tried with both names

@FastNinja
Copy link

@FastNinja FastNinja commented Aug 24, 2017

code coverage - this is the most basic requirement. what serious organisation will commit to using dot net core in non Windows environment if such a basic functionality is not there? how business can make a decision to release something in production where test coverage is not known?
It can be as fast and awesome - but without this basic support it pretty much unusable outside MS/Windows worlds.
The Linux community would like to embrace dot net core on board - but look it just simply can't.

@ollyjshaw
Copy link

@ollyjshaw ollyjshaw commented Aug 25, 2017

Yes, this feature should be available for all target platforms. It should not require an installation of Visual Studio.

@DavidParks8
Copy link
Member

@DavidParks8 DavidParks8 commented Sep 2, 2017

Code coverage used to be a reason to upgrade to vs enterprise, but the rise of free alternatives such as OpenCover has removed the value proposition. Thus, code coverage should be added to the dotnet cli on all platforms to keep developers from using the alternative products available.

@tracyhoward
Copy link

@tracyhoward tracyhoward commented Sep 14, 2017

@harshjain2 Is there a timeframe for when this will be available?
Add support for dotnet test --collect:"Code Coverage" #981

@pvlakshm
Copy link
Contributor

@pvlakshm pvlakshm commented Sep 17, 2017

Supporting CC with the dotnet test command line is presently not scheduled in our pans leading up to Nov 2017.
However, the following will be possible with .NET Core tests:

  1. you can vstest.console.exe command line runner and generate CC information (the CC data collectors are supported on .NET45).
  2. If you are working from within VS on tests targeting .NET Core, then you will need to add a reference to the Microsoft.CodeCoverage NuGet package to your test project, and update both the test and app-under-test projects to generate ‘Full’ debug info. Please see "Working with Code Coverage" here: https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md (note that this is only a temporary workaround).

We are eager to get to this once we finish addressing a few more scenarios that are already underway.

@arpit-nagar
Copy link

@arpit-nagar arpit-nagar commented Sep 18, 2017

@pvlakshm regarding point 2.
Is it possible to get CC on windows build environment which have Test agent, and don't have VS?

@pvlakshm
Copy link
Contributor

@pvlakshm pvlakshm commented Sep 19, 2017

@arpit-nagar, we are working towards that.

@henryJack
Copy link

@henryJack henryJack commented Oct 27, 2017

Code coverage is an essential part of CI. By all means, have a premium version available in a nice helpful way in VS but there should be, for example, a dotnet test --with-coverage=True --fail-under=95 type command for CI.

@ISemyanchuk
Copy link

@ISemyanchuk ISemyanchuk commented Oct 27, 2017

Hi, I was able to run vstest.console.exe with code coverage for .net core 2.0 project locally, but on build agent, with the same setup( i think it is same :)) I've receiving next error :
"Failed to initialize client proxy: could not connect to test process."

@pavelhorak pavelhorak modified the milestones: 16.7.0, 16.6.0 Mar 26, 2020
@syedhassaanahmed
Copy link
Member

@syedhassaanahmed syedhassaanahmed commented Apr 15, 2020

We need this functionality in our current project. Any update?

@jakubch1
Copy link
Member

@jakubch1 jakubch1 commented Apr 15, 2020

Did you try coverlet?

@syedhassaanahmed
Copy link
Member

@syedhassaanahmed syedhassaanahmed commented Apr 15, 2020

@jakubch1 No. Should this issue be closed and indicated that coverlet is the answer, as @danilobreda also pointed out?

@gremlm
Copy link

@gremlm gremlm commented Apr 15, 2020

Coverlet is not the answer. This feature should be compatible when executed at solution containing .Net Core and .Net Framework projects.
MS buils can build such solution, all projects.
dotnet - only projects related to .Net Core.
Coverlet only works for .Net Core, so again it is custom feature

@da-edwards
Copy link

@da-edwards da-edwards commented May 7, 2020

Hey @nohwnd , @jakubch1 , just wondering if there are any timelines to this one? Keen to make use of the Azure Devops pull request code coverage feature and I'd rather keep our build hosts as Linux but if it's not likely to be fixed soon, I'll move to Windows for the builds, at least in the short term.

Thanks

@PureKrome
Copy link

@PureKrome PureKrome commented May 27, 2020

I'm still confused to why people are complaining about lack of code coverage support for XPlat when it is already here and has been for a while.

For example, this repo uses CI (on AppVeyor ... not Azure DevOps) that does linux + codecov:

dotnet test -c $env:CONFIGURATION 
            -v minimal 
            --no-build 
            --collect:"XPlat Code Coverage" 
            --settings coverlet.runsettings 
            --results-directory './CodeCoverageResults'

All the coverlet docs explain this.

In that repo, I added the following:

  • coverlet.runsettings so I can be specific about what to EXCLUDE from coverage. For my personal self, I don't want coverage of any tests. I just want coverage of the actual code itself.
  • made sure my test projects included the coverlet dependency, which is now default included to any new .net core app "xunit project" in visual studio. If you're not using VS (which is me sometimes + many others, like using VS Code) ... then just manually include the nuget package in your test csproj.

As a bonus, I also (during my CI) upload the codecoverage results to an online codecoverage website. You can swap this out and instead use a code coverage reporting tool instead, on your CI machine.


TL;DR;

  • there is .net core code coverage for linux and windows
  • get familiar the CLI .net commands. e.g. dotnet test
  • code coverage can work on any build system which has dotnet core SDK installed. Your own computer, AppVeyor, Azure DevOps and i'm assuming GitHub Actions.

Now, here's an awesome cat to make you smile and positive:

@da-edwards
Copy link

@da-edwards da-edwards commented May 28, 2020

Hey @PureKrome ,

Just to be clear on the coverage, I'm very happy with using Coverlet, we've been using it for > 18 months now without issue. However, Azure DevOps doesn't support Coverlet for pull request diff code coverage policies, which we would find useful (blocking PRs with a diff code coverage < x%), as it's currently supported only on Windows. As our build infrastructure is Linux based, that poses a bit of an issue.

Thanks

@PureKrome
Copy link

@PureKrome PureKrome commented May 28, 2020

Hi @da-edwards - ok, let me see if I understand the issue, so appologies if i'm getting all of this wrong.

  • If codecov < x% (eg. 85%) then 'error' so the CI fails, which means the PR shouldn't be approved.
  • CI system is Azure DevOps.

If this is correct, then CodeCov has the ability to error when a "threshhold" is not met.
REF: CodeCov 'threshold' docs.

That said, the threshold attribute is for the 'msbuild' nuget (option #2). I prefer the codecov option #1 'vstest' nuget. Now, the msbuild nuget seems to work on linux though, but i've not done it nor can I speak for it.

Q: Could more be done?
A: Yep

Q: Is it still a bit painful?
A: Yep :(

Q: Warts and all, could it actually work?
A: Could! I've not tried it though but it might work? Maybe post a question in the codecov issues about 'Threshold' argument and option #1 nuget. Maybe it might be a key/value setting in a runsettings file? (I use runsettings files with my codecov + option #1022

HTH.

@da-edwards
Copy link

@da-edwards da-edwards commented May 28, 2020

Hey @PureKrome not quite, we're not (yet) interested in the overall code coverage, just on the diff, so if I was to update a method and refactor 4 lines but only 2 of those are covered with unit tests we would like to see a coverage of 50%, regardless of how many other unit tests have been added during the commit.

That's the (very specific) feature that's supported on Windows but not on Linux ... I think anyway :)

Does that make sense?

@PureKrome
Copy link

@PureKrome PureKrome commented May 28, 2020

Yep - it does.... but .. I've never heard of that?! TIL!!!

So do you mean this?

image

@da-edwards
Copy link

@da-edwards da-edwards commented May 28, 2020

That's the one ... did you manage that with Linux or is that on a Windows box?

Thanks

@PureKrome
Copy link

@PureKrome PureKrome commented May 28, 2020

Windows :( I've not tried on linux (yet). Maybe we can move this convo over to the Codecov repo and re-ask there? (ping me into the issue you create?)

@thiagocamargos
Copy link

@thiagocamargos thiagocamargos commented May 28, 2020

My two cents here: dotnet core and all its stack (vstest included) is supposed to be cross-platform and support linux.
It is a big issue if I have to change code (and build|release instructions/package management ARE code) if I decide to change platform and everyone tells me the framework /stack is x-plat.
That is misguiding.

When people here tells others to use coverlet instead vstest coverage for linux builds, you are also telling to use coverlet for windows and ignore the vstest feature. And if that is true, then this feature may be considered useless by those who appreciate a cross-platform framework. Thus, it is better to remove it from the application to minimize development complexity.

Telling people to use coverlet is only deprioritizing this issue, keeping it open forever and misguiding people into thinking this will be done. It has been almost 3 years since this was opened and there is still no clear answer from the core team. I have been following it and being notified since the beginning and I'm wondering if 3 years of a open issue is not enough yet.

@nohwnd , @jakubch1 and @AbhitejJohn Please do take this into consideration and give a clear answer. If coverlet should be the answer, please give a statement telling this, and making it clear that vstest IS NOT and WILL NOT be fully compatible with Linux. Also, a guide to cover this need (pun intended) in coverlet should be written/refered to. If Linux support is to be implement, please also make it clear.

just a quick ps: i've been reading this too.

@AceHack
Copy link

@AceHack AceHack commented Jun 4, 2020

I would love to hear an update on this issue, it's very confusing on status and plans.

@JohnGalt1717
Copy link

@JohnGalt1717 JohnGalt1717 commented Jun 4, 2020

Sorry I haven’t had time. Tomorrow morning I’ll try and do something to make it obvious.

Really I think the real solution is this:

microsoft/vscode#97919 (comment)

A real standardized test framework in vs code that can be hooked into to handle all of this stuff and give us parity with vs.net

Please thumbs up if you agree.

@pavelhorak
Copy link
Member

@pavelhorak pavelhorak commented Jun 8, 2020

We’ve been recently reviewing existing .NET code coverage solutions and decided to enhance our dynamic code coverage solution /also referred to as VS code coverage by some/ and make it cross platform. It will continue supporting code coverage for managed code on both .NET/.NET Core frameworks and for C++ on Windows. This would provide the community with a similar dynamic instrumentation solution, where only configured modules loaded at runtime are instrumented, as VS Code Coverage scenarios employed. As a proof of concept, @jakubch1 built a new prototype based on CLR Instrumentation engine profiler and made it run on Linux and Windows. We also plan to support non-binary report format(s) so it will be easily possible to display code coverage results, especially on Azure DevOps. As of now, we encourage community users continue using Coverlet as a cross platform code coverage solution. I hope to be able to share the timeline soon.

@PureKrome
Copy link

@PureKrome PureKrome commented Jun 9, 2020

@pavelhorak (and others in that team):

@kendrahavens
Copy link
Member

@kendrahavens kendrahavens commented Jun 9, 2020

To add, Coverlet continues to provide an excellent cross-platform code coverage solution. Bringing the built-in dynamic code coverage solution cross-platform won’t replace it since Coverlet will still be the best option for those who need static code coverage (aka ahead-of-time compilation), which is particularly useful for testing services. This effort is an attempt to satisfy the customers who have specifically asked for a dynamic (on-demand instrumentation) cross-platform code coverage solution.

@da-edwards
Copy link

@da-edwards da-edwards commented Jun 11, 2020

Great news! Thanks @pavelhorak and the whole team. Really appreciate your time and efforts on this.

@nohwnd nohwnd modified the milestones: 16.7.0, 16.8.0 Jun 16, 2020
@iyusuboff
Copy link

@iyusuboff iyusuboff commented Jun 22, 2020

Coming from java, even the simplest tasks are a problem in dotnot

@pavelhorak pavelhorak modified the milestones: 16.8.0, 16.9.0 Oct 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.