New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for dotnet test --collect:"Code Coverage" #981

Open
harshjain2 opened this Issue Aug 4, 2017 · 114 comments

Comments

Projects
None yet
@harshjain2
Contributor

harshjain2 commented Aug 4, 2017

Description

Add support for code coverage through dotnet test on Windows OS machine.
dotnet test --collect:"Code Coverage"

@harshjain2 harshjain2 added this to the 15.5 milestone Aug 4, 2017

@harshjain2 harshjain2 changed the title from Add support for dotnet test --collector:"Code Coverage" to Add support for dotnet test --collect:"Code Coverage" Aug 4, 2017

@arpit-nagar

This comment has been minimized.

Show comment
Hide comment
@arpit-nagar

arpit-nagar Aug 4, 2017

Do you have any plans for Linux ??

(edit from @codito: there are atleast 22 more votes in #579 (comment) for code coverage support in linux. Using this comment to keep track of linux support request)

arpit-nagar commented Aug 4, 2017

Do you have any plans for Linux ??

(edit from @codito: there are atleast 22 more votes in #579 (comment) for code coverage support in linux. Using this comment to keep track of linux support request)

@nulltoken

This comment has been minimized.

Show comment
Hide comment
@nulltoken

nulltoken Aug 4, 2017

Why the "having Visual Studio 2017 Enterprise installed" limitation? Is there any plan to lift it?

I believe collecting test code coverage during a CI build (with only the .NET Core SDK being deployed) would be a very valid scenario.

nulltoken commented Aug 4, 2017

Why the "having Visual Studio 2017 Enterprise installed" limitation? Is there any plan to lift it?

I believe collecting test code coverage during a CI build (with only the .NET Core SDK being deployed) would be a very valid scenario.

@ctro

This comment has been minimized.

Show comment
Hide comment
@ctro

ctro Aug 7, 2017

👍 I'd very much love to have working code coverage within the confines of .NET Core.

ctro commented Aug 7, 2017

👍 I'd very much love to have working code coverage within the confines of .NET Core.

@harshjain2

This comment has been minimized.

Show comment
Hide comment
@harshjain2

harshjain2 Aug 9, 2017

Contributor

@arpit-nagar test code coverage is VS enterprise and windows only so far, current plan is to get the same level of support in dotnet test (windows + VS enterprise). Mac/linux support isn't in our immediate backlog. /cc @pvlakshm @sbaid to consider the feedback for prioritization.

Contributor

harshjain2 commented Aug 9, 2017

@arpit-nagar test code coverage is VS enterprise and windows only so far, current plan is to get the same level of support in dotnet test (windows + VS enterprise). Mac/linux support isn't in our immediate backlog. /cc @pvlakshm @sbaid to consider the feedback for prioritization.

@arpit-nagar

This comment has been minimized.

Show comment
Hide comment
@arpit-nagar

arpit-nagar Aug 9, 2017

@harshjain2 This is a very basic requirement. And as dotnet core is multi platform this feature should also need to be available on Linux.
Is there any way we can achieve code coverage on Linux for dotnet core project.

arpit-nagar commented Aug 9, 2017

@harshjain2 This is a very basic requirement. And as dotnet core is multi platform this feature should also need to be available on Linux.
Is there any way we can achieve code coverage on Linux for dotnet core project.

@andyjansson

This comment has been minimized.

Show comment
Hide comment
@andyjansson

andyjansson Aug 9, 2017

test code coverage is VS enterprise and windows only so far, current plan is to get the same level of support in dotnet test (windows + VS enterprise).

@harshjain2 What's the rationale behind requiring VS enterprise for code coverage?

andyjansson commented Aug 9, 2017

test code coverage is VS enterprise and windows only so far, current plan is to get the same level of support in dotnet test (windows + VS enterprise).

@harshjain2 What's the rationale behind requiring VS enterprise for code coverage?

@rol-dave-overall

This comment has been minimized.

Show comment
Hide comment
@rol-dave-overall

rol-dave-overall Aug 15, 2017

Downloaded VS2017 15.3 and .NET Core 2.0 and after entering (dotnet test --collect:"Code Coverage") I get an error:

The test source file "C:\Application2\tests\Console.UnitTests1\Coverage" provided was not found.

Tried changing this to be (dotnet test --collect:"CodeCoverage") i.e. without the space and then get this error:

dotnet exec needs a managed .dll or .exe extension. The application specified was 'C:\Program'
Object reference not set to an instance of an object.

rol-dave-overall commented Aug 15, 2017

Downloaded VS2017 15.3 and .NET Core 2.0 and after entering (dotnet test --collect:"Code Coverage") I get an error:

The test source file "C:\Application2\tests\Console.UnitTests1\Coverage" provided was not found.

Tried changing this to be (dotnet test --collect:"CodeCoverage") i.e. without the space and then get this error:

dotnet exec needs a managed .dll or .exe extension. The application specified was 'C:\Program'
Object reference not set to an instance of an object.

@jstallm

This comment has been minimized.

Show comment
Hide comment
@jstallm

jstallm Aug 18, 2017

It sounds to me like the status is that we cannot do code coverage for asp.net core 1.X using dotnet test at the command line. Is this correct?

jstallm commented Aug 18, 2017

It sounds to me like the status is that we cannot do code coverage for asp.net core 1.X using dotnet test at the command line. Is this correct?

@edumserrano

This comment has been minimized.

Show comment
Hide comment
@edumserrano

edumserrano Aug 21, 2017

I get the same as @rol-dave-overall. This is a show-stopper for us trying to get our solution build on Visual Studio Team Services. Disabling code coverage is not really an option.

Any available workarounds?

edumserrano commented Aug 21, 2017

I get the same as @rol-dave-overall. This is a show-stopper for us trying to get our solution build on Visual Studio Team Services. Disabling code coverage is not really an option.

Any available workarounds?

@RomanKernSW

This comment has been minimized.

Show comment
Hide comment
@RomanKernSW

RomanKernSW Aug 23, 2017

actually have the same problem. the vso test with --collect Coverage or "Code Coverage" not working. the runsettings tried with both names

RomanKernSW commented Aug 23, 2017

actually have the same problem. the vso test with --collect Coverage or "Code Coverage" not working. the runsettings tried with both names

@FastNinja

This comment has been minimized.

Show comment
Hide comment
@FastNinja

FastNinja Aug 24, 2017

code coverage - this is the most basic requirement. what serious organisation will commit to using dot net core in non Windows environment if such a basic functionality is not there? how business can make a decision to release something in production where test coverage is not known?
It can be as fast and awesome - but without this basic support it pretty much unusable outside MS/Windows worlds.
The Linux community would like to embrace dot net core on board - but look it just simply can't.

FastNinja commented Aug 24, 2017

code coverage - this is the most basic requirement. what serious organisation will commit to using dot net core in non Windows environment if such a basic functionality is not there? how business can make a decision to release something in production where test coverage is not known?
It can be as fast and awesome - but without this basic support it pretty much unusable outside MS/Windows worlds.
The Linux community would like to embrace dot net core on board - but look it just simply can't.

@ollyjshaw

This comment has been minimized.

Show comment
Hide comment
@ollyjshaw

ollyjshaw Aug 25, 2017

Yes, this feature should be available for all target platforms. It should not require an installation of Visual Studio.

ollyjshaw commented Aug 25, 2017

Yes, this feature should be available for all target platforms. It should not require an installation of Visual Studio.

@DavidParks8

This comment has been minimized.

Show comment
Hide comment
@DavidParks8

DavidParks8 Sep 2, 2017

Member

Code coverage used to be a reason to upgrade to vs enterprise, but the rise of free alternatives such as OpenCover has removed the value proposition. Thus, code coverage should be added to the dotnet cli on all platforms to keep developers from using the alternative products available.

Member

DavidParks8 commented Sep 2, 2017

Code coverage used to be a reason to upgrade to vs enterprise, but the rise of free alternatives such as OpenCover has removed the value proposition. Thus, code coverage should be added to the dotnet cli on all platforms to keep developers from using the alternative products available.

@tracyhoward

This comment has been minimized.

Show comment
Hide comment
@tracyhoward

tracyhoward Sep 14, 2017

@harshjain2 Is there a timeframe for when this will be available?
Add support for dotnet test --collect:"Code Coverage" #981

tracyhoward commented Sep 14, 2017

@harshjain2 Is there a timeframe for when this will be available?
Add support for dotnet test --collect:"Code Coverage" #981

@pvlakshm

This comment has been minimized.

Show comment
Hide comment
@pvlakshm

pvlakshm Sep 17, 2017

Member

Supporting CC with the dotnet test command line is presently not scheduled in our pans leading up to Nov 2017.
However, the following will be possible with .NET Core tests:

  1. you can vstest.console.exe command line runner and generate CC information (the CC data collectors are supported on .NET45).
  2. If you are working from within VS on tests targeting .NET Core, then you will need to add a reference to the Microsoft.CodeCoverage NuGet package to your test project, and update both the test and app-under-test projects to generate ‘Full’ debug info. Please see "Working with Code Coverage" here: https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md (note that this is only a temporary workaround).

We are eager to get to this once we finish addressing a few more scenarios that are already underway.

Member

pvlakshm commented Sep 17, 2017

Supporting CC with the dotnet test command line is presently not scheduled in our pans leading up to Nov 2017.
However, the following will be possible with .NET Core tests:

  1. you can vstest.console.exe command line runner and generate CC information (the CC data collectors are supported on .NET45).
  2. If you are working from within VS on tests targeting .NET Core, then you will need to add a reference to the Microsoft.CodeCoverage NuGet package to your test project, and update both the test and app-under-test projects to generate ‘Full’ debug info. Please see "Working with Code Coverage" here: https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md (note that this is only a temporary workaround).

We are eager to get to this once we finish addressing a few more scenarios that are already underway.

@arpit-nagar

This comment has been minimized.

Show comment
Hide comment
@arpit-nagar

arpit-nagar Sep 18, 2017

@pvlakshm regarding point 2.
Is it possible to get CC on windows build environment which have Test agent, and don't have VS?

arpit-nagar commented Sep 18, 2017

@pvlakshm regarding point 2.
Is it possible to get CC on windows build environment which have Test agent, and don't have VS?

@pvlakshm

This comment has been minimized.

Show comment
Hide comment
@pvlakshm

pvlakshm Sep 19, 2017

Member

@arpit-nagar, we are working towards that.

Member

pvlakshm commented Sep 19, 2017

@arpit-nagar, we are working towards that.

@henryJack

This comment has been minimized.

Show comment
Hide comment
@henryJack

henryJack Oct 27, 2017

Code coverage is an essential part of CI. By all means, have a premium version available in a nice helpful way in VS but there should be, for example, a dotnet test --with-coverage=True --fail-under=95 type command for CI.

henryJack commented Oct 27, 2017

Code coverage is an essential part of CI. By all means, have a premium version available in a nice helpful way in VS but there should be, for example, a dotnet test --with-coverage=True --fail-under=95 type command for CI.

@ISemyanchuk

This comment has been minimized.

Show comment
Hide comment
@ISemyanchuk

ISemyanchuk Oct 27, 2017

Hi, I was able to run vstest.console.exe with code coverage for .net core 2.0 project locally, but on build agent, with the same setup( i think it is same :)) I've receiving next error :
"Failed to initialize client proxy: could not connect to test process."

ISemyanchuk commented Oct 27, 2017

Hi, I was able to run vstest.console.exe with code coverage for .net core 2.0 project locally, but on build agent, with the same setup( i think it is same :)) I've receiving next error :
"Failed to initialize client proxy: could not connect to test process."

@SharpNoiZy

This comment has been minimized.

Show comment
Hide comment
@SharpNoiZy

SharpNoiZy Aug 2, 2018

Please support the ExcludeFromCodeCoverage Attribute.

Except that, --logger "trx" --collect "Code Coverage" works with the new SDK.

SharpNoiZy commented Aug 2, 2018

Please support the ExcludeFromCodeCoverage Attribute.

Except that, --logger "trx" --collect "Code Coverage" works with the new SDK.

@smadala

This comment has been minimized.

Show comment
Hide comment
@smadala

smadala Aug 2, 2018

Member

Please support the ExcludeFromCodeCoverage Attribute.
Except that, --logger "trx" --collect "Code Coverage" works with the new SDK.

Related: #1548

/cc: @pvlakshm @cltshivash @mayankbansal018

Member

smadala commented Aug 2, 2018

Please support the ExcludeFromCodeCoverage Attribute.
Except that, --logger "trx" --collect "Code Coverage" works with the new SDK.

Related: #1548

/cc: @pvlakshm @cltshivash @mayankbansal018

@onovotny

This comment has been minimized.

Show comment
Hide comment
@onovotny

onovotny Aug 2, 2018

Please do not close this issue. This issue should remain open until embedded PDB's are supported.

onovotny commented Aug 2, 2018

Please do not close this issue. This issue should remain open until embedded PDB's are supported.

@henryJack

This comment has been minimized.

Show comment
Hide comment
@henryJack

henryJack Aug 2, 2018

@livarcocc - This is still an issue and should not be closed

henryJack commented Aug 2, 2018

@livarcocc - This is still an issue and should not be closed

@livarcocc

This comment has been minimized.

Show comment
Hide comment
@livarcocc

livarcocc Aug 2, 2018

Member

I didn't really close this issue. github did when @smadala made a PR tagged as fixing it. @smadala can you re-activate it?

Member

livarcocc commented Aug 2, 2018

I didn't really close this issue. github did when @smadala made a PR tagged as fixing it. @smadala can you re-activate it?

@Larswa

This comment has been minimized.

Show comment
Hide comment
@Larswa

Larswa Aug 21, 2018

@mpetito Codecoverage comes with the nuget package, and it comes with the VSTS "Visual Studio Test Platform Installer" task.
In my case I end up with this on my buildserver that doesn't have VS2017 installed btw ...
D:\VSTSAgent\_work\_tool\VsTest\15.8.0\x64\tools\net451\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe

Larswa commented Aug 21, 2018

@mpetito Codecoverage comes with the nuget package, and it comes with the VSTS "Visual Studio Test Platform Installer" task.
In my case I end up with this on my buildserver that doesn't have VS2017 installed btw ...
D:\VSTSAgent\_work\_tool\VsTest\15.8.0\x64\tools\net451\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe

@vikasillumina

This comment has been minimized.

Show comment
Hide comment
@vikasillumina

vikasillumina Aug 25, 2018

@harshjain2 any timelines for providing this support? I am currently working on building the CI/CD for our .NET core based projects and we use docker for running the CI/CD and this feature is missing. If code coverage is natively supported with dotnet test command itself it will make it very easy to use this.

vikasillumina commented Aug 25, 2018

@harshjain2 any timelines for providing this support? I am currently working on building the CI/CD for our .NET core based projects and we use docker for running the CI/CD and this feature is missing. If code coverage is natively supported with dotnet test command itself it will make it very easy to use this.

@mpetito

This comment has been minimized.

Show comment
Hide comment
@mpetito

mpetito Aug 25, 2018

@vikasillumina use coverlet for now.

I ended up using coverlet even in Windows build environments as the workarounds required above are less than ideal. The issues I ran into included:

  • Output of 0 byte .coverage files without any warning or explanation.
  • Binary coverage outputs that I couldn't use otherwise.
  • Separate package references needed to download CodeCoverage.exe to run binary-to-xml conversion.

After spending hours trying to get this to work, I added coverlet and had my coverage imported into sonarqube in about 10 minutes.

mpetito commented Aug 25, 2018

@vikasillumina use coverlet for now.

I ended up using coverlet even in Windows build environments as the workarounds required above are less than ideal. The issues I ran into included:

  • Output of 0 byte .coverage files without any warning or explanation.
  • Binary coverage outputs that I couldn't use otherwise.
  • Separate package references needed to download CodeCoverage.exe to run binary-to-xml conversion.

After spending hours trying to get this to work, I added coverlet and had my coverage imported into sonarqube in about 10 minutes.

@li0nsar3c00l

This comment has been minimized.

Show comment
Hide comment
@li0nsar3c00l

li0nsar3c00l Aug 29, 2018

@vikasillumina 2.1.400 is not in preview anymore, so the feature exist and can be used. I got it working in my pipeline. only downside is, that you only get the coverage report and not an output of the percentage.

li0nsar3c00l commented Aug 29, 2018

@vikasillumina 2.1.400 is not in preview anymore, so the feature exist and can be used. I got it working in my pipeline. only downside is, that you only get the coverage report and not an output of the percentage.

@JohnGalt1717

This comment has been minimized.

Show comment
Hide comment
@JohnGalt1717

JohnGalt1717 Aug 29, 2018

@li0nsar3c00l Does this work on Linux now?

JohnGalt1717 commented Aug 29, 2018

@li0nsar3c00l Does this work on Linux now?

@joymon

This comment has been minimized.

Show comment
Hide comment
@joymon

joymon Aug 29, 2018

@codito ,
I tried to install the Microsoft.TestPlatform nuget package to .Net standard test project. But the codecoverage.exe didn't showed up. Any ideas other than checking in the codecoverage.exe with in source

joymon commented Aug 29, 2018

@codito ,
I tried to install the Microsoft.TestPlatform nuget package to .Net standard test project. But the codecoverage.exe didn't showed up. Any ideas other than checking in the codecoverage.exe with in source

@vikasillumina

This comment has been minimized.

Show comment
Hide comment
@vikasillumina

vikasillumina Aug 29, 2018

@li0nsar3c00l I actually tried the 2.1.400 version and coverage option does work, however its only supported on Windows. Please correct me if I am wrong. I am looking to use the same command natively on docker running linux. :(
Thanks @mpetito for your inputs and lessons learnt on the way to use coverlet, I will give it a try.

vikasillumina commented Aug 29, 2018

@li0nsar3c00l I actually tried the 2.1.400 version and coverage option does work, however its only supported on Windows. Please correct me if I am wrong. I am looking to use the same command natively on docker running linux. :(
Thanks @mpetito for your inputs and lessons learnt on the way to use coverlet, I will give it a try.

@codito

This comment has been minimized.

Show comment
Hide comment
@codito

codito Aug 30, 2018

Contributor

@joymon it seems to work for me, here's what I tried:

# create a new netstandard2.0 class lib
> dotnet new classlib
> dotnet add package Microsoft.TestPlatform
# see the csproj for classlib, it should have Microsoft.TestPlatform added now
> dotnet restore
> ls ~/.nuget/packages/microsoft.testplatform/15.8.0/tools/net451/Team\ Tools/Dynamic\ Code\ Coverage\ Tools/CodeCoverage.exe
# ~/.nuget is the default nuget package cache on my box. See global-packages details in https://docs.microsoft.com/en-us/nuget/consume-packages/managing-the-global-packages-and-cache-folders

Please don't check-in the exe in any case :) As an alternative, consider adding a step in your CI to download the nuget package directly. Like this in powershell script:

> Invoke-WebRequest "https://www.nuget.org/api/v2/package/Microsoft.TestPlatform" -OutFile Microsoft.TestPlatform.nupkg
> Expand-Archive Microsoft.TestPlatform.nupkg -DestinationPath ./test-pkg
Contributor

codito commented Aug 30, 2018

@joymon it seems to work for me, here's what I tried:

# create a new netstandard2.0 class lib
> dotnet new classlib
> dotnet add package Microsoft.TestPlatform
# see the csproj for classlib, it should have Microsoft.TestPlatform added now
> dotnet restore
> ls ~/.nuget/packages/microsoft.testplatform/15.8.0/tools/net451/Team\ Tools/Dynamic\ Code\ Coverage\ Tools/CodeCoverage.exe
# ~/.nuget is the default nuget package cache on my box. See global-packages details in https://docs.microsoft.com/en-us/nuget/consume-packages/managing-the-global-packages-and-cache-folders

Please don't check-in the exe in any case :) As an alternative, consider adding a step in your CI to download the nuget package directly. Like this in powershell script:

> Invoke-WebRequest "https://www.nuget.org/api/v2/package/Microsoft.TestPlatform" -OutFile Microsoft.TestPlatform.nupkg
> Expand-Archive Microsoft.TestPlatform.nupkg -DestinationPath ./test-pkg
@vikasillumina

This comment has been minimized.

Show comment
Hide comment
@vikasillumina

vikasillumina Aug 30, 2018

@mpetito Thanks once again for your input and I was able to get the Coverlet to work without any issues other than creating the Result directory to save the output file on Windows (few team members use windows for local development) On mac it worked like charm! I didn't need to do any conversion just used this command:
dotnet test %csproj file% /p:CollectCoverage=true /p:CoverletOutput='./Results/' /p:CoverletOutputFormat=opencover and I got the report in xml format without any conversion needed. Please note I used 2.2.1 version of the coverlet.msbuild library https://www.nuget.org/packages/coverlet.msbuild/2.2.1 in case it made the difference.

I tried opencover and cobertura and both options worked fine for me.

vikasillumina commented Aug 30, 2018

@mpetito Thanks once again for your input and I was able to get the Coverlet to work without any issues other than creating the Result directory to save the output file on Windows (few team members use windows for local development) On mac it worked like charm! I didn't need to do any conversion just used this command:
dotnet test %csproj file% /p:CollectCoverage=true /p:CoverletOutput='./Results/' /p:CoverletOutputFormat=opencover and I got the report in xml format without any conversion needed. Please note I used 2.2.1 version of the coverlet.msbuild library https://www.nuget.org/packages/coverlet.msbuild/2.2.1 in case it made the difference.

I tried opencover and cobertura and both options worked fine for me.

@StuffOfInterest

This comment has been minimized.

Show comment
Hide comment
@StuffOfInterest

StuffOfInterest Sep 13, 2018

Even with the 15.8 tools there appears to still be issues with code coverage running under some scenarios. I was able to get the command line to work on my local PC. When I tried adding it to a TFS build things didn't go so well. The command that ran was:

"C:\Program Files\dotnet\dotnet.exe" test B:\agent\_work\43\s\Something.Web.Tests\Something.Web.Tests.csproj --no-build --verbosity normal --collect "Code Coverage" --logger trx --results-directory B:\agent\_work\_temp

What came out was many, many instances of errors like this:

Failed   Index_view_convention_sets_ActivityName_to_View_All_XYZ
Error Message:
 Assembly Initialization method Something.Web.Tests.AssemblyInitialization.InitalizeAppConfiguration threw exception. System.OutOfMemoryException: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.. Aborting test execution.
Stack Trace:
    at Something.Web.Tests.AssemblyInitialization.InitalizeAppConfiguration(TestContext context)

This is being run on a build server with TFS on premises. The build server has Visual Studio 2017 Enterprise version 15.8 installed and the project being tests has the Microsoft.NET.Test.Sdk version 15.8.0 installed in it. If the "--collect Code Coverage" is removed the test run works fine.

StuffOfInterest commented Sep 13, 2018

Even with the 15.8 tools there appears to still be issues with code coverage running under some scenarios. I was able to get the command line to work on my local PC. When I tried adding it to a TFS build things didn't go so well. The command that ran was:

"C:\Program Files\dotnet\dotnet.exe" test B:\agent\_work\43\s\Something.Web.Tests\Something.Web.Tests.csproj --no-build --verbosity normal --collect "Code Coverage" --logger trx --results-directory B:\agent\_work\_temp

What came out was many, many instances of errors like this:

Failed   Index_view_convention_sets_ActivityName_to_View_All_XYZ
Error Message:
 Assembly Initialization method Something.Web.Tests.AssemblyInitialization.InitalizeAppConfiguration threw exception. System.OutOfMemoryException: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.. Aborting test execution.
Stack Trace:
    at Something.Web.Tests.AssemblyInitialization.InitalizeAppConfiguration(TestContext context)

This is being run on a build server with TFS on premises. The build server has Visual Studio 2017 Enterprise version 15.8 installed and the project being tests has the Microsoft.NET.Test.Sdk version 15.8.0 installed in it. If the "--collect Code Coverage" is removed the test run works fine.

@mayankbansal018

This comment has been minimized.

Show comment
Hide comment
@mayankbansal018

mayankbansal018 Sep 17, 2018

Contributor

@StuffOfInterest , we have seen similar issue, where OOM exception is thrown when Coverage is enabled.
To validate that your issue is similar to one we fixed internally, can you please provide following info

  • Are you facing this issue consistently with Coverage on, or is it intermediate?
  • Are your tests netcore of full CLR based?
  • When you run things locally, can you please take a quick look, & check what is the approx peak memory consumption of "testhost(if fullCLR tests)" else check "child dotnet.exe(for netcore tests)" for process. Hoping your tests don't take too much time to run.
Contributor

mayankbansal018 commented Sep 17, 2018

@StuffOfInterest , we have seen similar issue, where OOM exception is thrown when Coverage is enabled.
To validate that your issue is similar to one we fixed internally, can you please provide following info

  • Are you facing this issue consistently with Coverage on, or is it intermediate?
  • Are your tests netcore of full CLR based?
  • When you run things locally, can you please take a quick look, & check what is the approx peak memory consumption of "testhost(if fullCLR tests)" else check "child dotnet.exe(for netcore tests)" for process. Hoping your tests don't take too much time to run.
@StuffOfInterest

This comment has been minimized.

Show comment
Hide comment
@StuffOfInterest

StuffOfInterest Sep 17, 2018

@mayankbansal018 , I'm seeing the issue very consistently. All is happening with netcore. The memory footprint is not excessive.

First, a bit of background. I'm converting over an existing code base of libraries from .NET Framework to .NET Standard. There are currently six library projects. Each library project has a unit test project targeting .NET Core 2.1. Across the six test projects there are just over 1000 unit tests in total, which unfortunately only provide about 50% code coverage.

The .NET build step in the TFS Build service runs "dotnet test" individually for each unit test project. One of the test projects only has eight unit tests in it right now. To make the verification as easy as possible I setup a build to only run testing for this one project.

Both "dotnet build" and "dotnet restore" ran fine. When it came time for "dotnet test" to run the OOM exceptions happened for all eight tests. I had Task Manager running on the build server while the build was running. No processes went over 250 MB of memory consumption while this was happening.

The test project has version 15.8.0 of the Microsoft.NET.Test.Sdk referenced. I've installed version 2.1.402 of the .NET Core SDK on the build server. Version 15.8.0 of Visual Studio Enterprise is installed on the test server. Later this morning I plan to update Visual Studio to 15.8.4, but this will take some work being that the build server does not have direct Internet access.

StuffOfInterest commented Sep 17, 2018

@mayankbansal018 , I'm seeing the issue very consistently. All is happening with netcore. The memory footprint is not excessive.

First, a bit of background. I'm converting over an existing code base of libraries from .NET Framework to .NET Standard. There are currently six library projects. Each library project has a unit test project targeting .NET Core 2.1. Across the six test projects there are just over 1000 unit tests in total, which unfortunately only provide about 50% code coverage.

The .NET build step in the TFS Build service runs "dotnet test" individually for each unit test project. One of the test projects only has eight unit tests in it right now. To make the verification as easy as possible I setup a build to only run testing for this one project.

Both "dotnet build" and "dotnet restore" ran fine. When it came time for "dotnet test" to run the OOM exceptions happened for all eight tests. I had Task Manager running on the build server while the build was running. No processes went over 250 MB of memory consumption while this was happening.

The test project has version 15.8.0 of the Microsoft.NET.Test.Sdk referenced. I've installed version 2.1.402 of the .NET Core SDK on the build server. Version 15.8.0 of Visual Studio Enterprise is installed on the test server. Later this morning I plan to update Visual Studio to 15.8.4, but this will take some work being that the build server does not have direct Internet access.

@mayankbansal018

This comment has been minimized.

Show comment
Hide comment
@mayankbansal018

mayankbansal018 Oct 16, 2018

Contributor

@StuffOfInterest , apologies for getting so late on this. Can you please create a separate bug for your issue, & see if you can attach a repro project to it.
At the very minimum please do attach the diagnostics logs that you get when you generate Code Coverage for your projects.

Contributor

mayankbansal018 commented Oct 16, 2018

@StuffOfInterest , apologies for getting so late on this. Can you please create a separate bug for your issue, & see if you can attach a repro project to it.
At the very minimum please do attach the diagnostics logs that you get when you generate Code Coverage for your projects.

@StuffOfInterest

This comment has been minimized.

Show comment
Hide comment
@StuffOfInterest

StuffOfInterest Oct 16, 2018

@mayankbansal018 , it looks like that may not be necessary. I updated to the recently released Microsoft.NET.Test.Sdk version 15.9.0 and updated the build server's Visual Studio to 15.8.6. Now, the tests are working!

image

One of those two appears to have solved the memory issue.

Thanks.

StuffOfInterest commented Oct 16, 2018

@mayankbansal018 , it looks like that may not be necessary. I updated to the recently released Microsoft.NET.Test.Sdk version 15.9.0 and updated the build server's Visual Studio to 15.8.6. Now, the tests are working!

image

One of those two appears to have solved the memory issue.

Thanks.

@mayankbansal018

This comment has been minimized.

Show comment
Hide comment
@mayankbansal018

mayankbansal018 Oct 16, 2018

Contributor

@StuffOfInterest , great to see it got resolved. 😄

Contributor

mayankbansal018 commented Oct 16, 2018

@StuffOfInterest , great to see it got resolved. 😄

@JohnGalt1717

This comment has been minimized.

Show comment
Hide comment
@JohnGalt1717

JohnGalt1717 Oct 16, 2018

Still requires Visual studio and isn't a dotnet.exe function so it doesn't work in docker for instance. And using coverlet etc. is VERY slow and often fails.

This needs to be independent of Visual Studio and just work as part of dotnet.exe

JohnGalt1717 commented Oct 16, 2018

Still requires Visual studio and isn't a dotnet.exe function so it doesn't work in docker for instance. And using coverlet etc. is VERY slow and often fails.

This needs to be independent of Visual Studio and just work as part of dotnet.exe

@henryJack

This comment has been minimized.

Show comment
Hide comment
@henryJack

henryJack Oct 16, 2018

Still desperately needed for linux/Docker and is the only thing standing in the way before my corporation adopts .NET core.

henryJack commented Oct 16, 2018

Still desperately needed for linux/Docker and is the only thing standing in the way before my corporation adopts .NET core.

@carlosrpg

This comment has been minimized.

Show comment
Hide comment
@carlosrpg

carlosrpg Oct 16, 2018

Not resolved to me, If enable code coverage I get the following error: (I'm using on TFS)
2018-10-16T20:56:55.6793227Z Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: Failed to initialize client proxy: could not connect to test process. 2018-10-16T20:56:55.6793768Z at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable1 sources) 2018-10-16T20:56:55.6794376Z at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.StartTestRun(TestRunCriteria testRunCriteria, ITestRunEventsHandler eventHandler)

  • I have the <DebugType>Full</DebugType> o both test and test project
  • I have the nuget for <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" /> on the test project
  • I Have Microsoft.TestPlatform -Version 15.9.0
  • the vstest command is: [command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" MyTestDLL.dll /EnableCodeCoverage /logger:trx /framework:.NETCoreApp,Version=v2.0

It works fine locally but not on the TFS.

If I remove the /EnableCodeCoverage it runs all the tests just fine

carlosrpg commented Oct 16, 2018

Not resolved to me, If enable code coverage I get the following error: (I'm using on TFS)
2018-10-16T20:56:55.6793227Z Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: Failed to initialize client proxy: could not connect to test process. 2018-10-16T20:56:55.6793768Z at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable1 sources) 2018-10-16T20:56:55.6794376Z at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.StartTestRun(TestRunCriteria testRunCriteria, ITestRunEventsHandler eventHandler)

  • I have the <DebugType>Full</DebugType> o both test and test project
  • I have the nuget for <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" /> on the test project
  • I Have Microsoft.TestPlatform -Version 15.9.0
  • the vstest command is: [command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" MyTestDLL.dll /EnableCodeCoverage /logger:trx /framework:.NETCoreApp,Version=v2.0

It works fine locally but not on the TFS.

If I remove the /EnableCodeCoverage it runs all the tests just fine

@mayankbansal018

This comment has been minimized.

Show comment
Hide comment
@mayankbansal018

mayankbansal018 Oct 17, 2018

Contributor

@JohnGalt1717 , the coverage support is not longer tied to Visual Studio, & is available with dotnet.exe. Can you please try it again with latest dotnet.exe, & make sure you also have latest version of Microsoft.Net.Test.SDK. Please let me know if you face any errors with it.

@henryJack As mentioned previously, it is part of our black log, we are exploring what all gaps we need to fill, to move our current implementation from windows only, & take it x-plat.

@carlosrpg, the error you have shared seems unrelated to code coverage. Can you please try setting env variable VSTEST_CONNECTION_TIMEOUT=200, & try again, also please enable diagnostics logs via --diag:<path_to_somefile>, & share those with us.

Contributor

mayankbansal018 commented Oct 17, 2018

@JohnGalt1717 , the coverage support is not longer tied to Visual Studio, & is available with dotnet.exe. Can you please try it again with latest dotnet.exe, & make sure you also have latest version of Microsoft.Net.Test.SDK. Please let me know if you face any errors with it.

@henryJack As mentioned previously, it is part of our black log, we are exploring what all gaps we need to fill, to move our current implementation from windows only, & take it x-plat.

@carlosrpg, the error you have shared seems unrelated to code coverage. Can you please try setting env variable VSTEST_CONNECTION_TIMEOUT=200, & try again, also please enable diagnostics logs via --diag:<path_to_somefile>, & share those with us.

@JohnGalt1717

This comment has been minimized.

Show comment
Hide comment
@JohnGalt1717

JohnGalt1717 Oct 17, 2018

@mayankbansal018 I can't find any documentation on how this is supposed to be done. We're using VSTS (Azure pipelines) and docker. I'm not sure what the output is and how to get it and have it run.

Here's my entrypoint:

ENTRYPOINT dotnet test /src/CADLearning.Api.Test/CADLearning.Api.Test.csproj -c Release --no-restore --collect "Code Coverage" --logger trx --results-directory /var/results

Which results in this in the logs:

Data collector 'Code Coverage' message: No code coverage data available. Code coverage is currently supported only on Windows..

Which is the point. This needs to work with dotnet.exe on Linux for this to work with Docker properly. I can't be running tests on windows when the code is running on a Linux docker container for obvious reasons.

This is a major issue that needs to be escalated and fixed so that this works right.

JohnGalt1717 commented Oct 17, 2018

@mayankbansal018 I can't find any documentation on how this is supposed to be done. We're using VSTS (Azure pipelines) and docker. I'm not sure what the output is and how to get it and have it run.

Here's my entrypoint:

ENTRYPOINT dotnet test /src/CADLearning.Api.Test/CADLearning.Api.Test.csproj -c Release --no-restore --collect "Code Coverage" --logger trx --results-directory /var/results

Which results in this in the logs:

Data collector 'Code Coverage' message: No code coverage data available. Code coverage is currently supported only on Windows..

Which is the point. This needs to work with dotnet.exe on Linux for this to work with Docker properly. I can't be running tests on windows when the code is running on a Linux docker container for obvious reasons.

This is a major issue that needs to be escalated and fixed so that this works right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment