Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverlet freezes test execution #765

Closed
NPadrutt opened this issue Mar 17, 2020 · 10 comments
Closed

Coverlet freezes test execution #765

NPadrutt opened this issue Mar 17, 2020 · 10 comments
Labels
needs more info More details are needed tenet-performance Performance related issue

Comments

@NPadrutt
Copy link

I have a Testset of around 10'000 Tests which I execute with the following command:

dotnet test ASK.UnitTests.sln --configuration Debug --no-build --no-build -l "console;verbosity=normal" -a C:\Users\padruttn\Documents\git\ask\packages\xunit.runner.visualstudio.2.4.1\build\_common

Now I wanted to add code coverage to that so I modified the command to the following:

dotnet test ASK.UnitTests.sln --configuration Debug --no-build --no-build -l "console;verbosity=normal" -a C:\Users\padruttn\Documents\git\ask\packages\xunit.runner.visualstudio.2.4.1\build\_common /p:CollectCoverage=true /p:CoverletOutputFormat=opencover

With that the test execution grinds to a halt after a couple of hundert tests and doesn't make any move (I wated on the build server for 2h+ and it didn't recover). When I set "CollectCoverage" to false all tests run through again.
At one point I was able to reproduce the issue locally. When I tried to cancel the run I got this warning which might be related:
warning MSB4220: Waiting for the currently executing task "Coverlet.MSbuild.Tasks.InstrumentationTask" to cancel.
That Warning is just repeated all 10 seconds again. I only waited 5 minutes till I went on killing the .net core host, but it didn't seem to stop soon.
Any idea why that is and how I can fix that?

@NPadrutt
Copy link
Author

I should maybe add: during this time the whole server is spinning on 100% CPU for the whole time. SO not sure if there is a loop or something here?

@MarcoRossignoli
Copy link
Collaborator

MarcoRossignoli commented Mar 18, 2020

First of all why not simply?
dotnet test ASK.UnitTests.sln --configuration Debug --no-build -l "console;verbosity=normal"
Is that adapter path mandatory?

BTW

Interesting...never seen something like that, can you please try to enable only 1 test for every project test once a time I mean run coverage for one csproj test at time, we need to understand if you've some code that lead coverlet instrumentation loops, it's very strange and I'm pretty sure we don't have any loop in code.
Another possible issue could be a simple performance fact, you stopped run and that message simply saying that at that time it was inside instrumenting but dotnet test *.sln runs parallel build/test so there is possibility that simply test are slow related to fact that coverlet instruments code and inject a method call to register hit, so in some cases tests with a lot of loops slow down a lot tests with coverage because block some jitter optimization and also simply because more code must be executed.
Take look here #511 (comment) every RecordSingleHit is injected by coverlet and it's a code to run.

Could be great if you profile process like #511 (comment) with some tool(VS in this case because I'm on win) so we can see where time is spent.

@MarcoRossignoli MarcoRossignoli added needs more info More details are needed tenet-performance Performance related issue labels Mar 18, 2020
@NPadrutt
Copy link
Author

Fair question ;) I think I can scrap that one. I saw that in a tutorial and I wasn't sure at one point if xUnit Tests are discovered correctly so I added that one as well. But that should be optional and in that context not needed.

I just tried /p:SingleHit=true and it seems to do the trick! With that it runs through completly :) thank you very much :)

As far as I understand it, that just means that every line is only counted once, right? Is there a reason why that is not on per default?

@MarcoRossignoli
Copy link
Collaborator

MarcoRossignoli commented Mar 18, 2020

As far as I understand it, that just means that every line is only counted once, right? Is there a reason why that is not on per default?

Yes is counted only one time so no register/cache update in processor so faster run, it's not a default because by default the number of time you pass through a line or a branch is an index of "good coverage" I mean more test cover that line better was(should clearly this is only a number) the testing.

You could understand which project is so slow and pass single hit only to that to have better number...but it depends on your strategy and on the kind of coverage you want.

Feel free to close this issue if resolved!

Thank's a lot for using coverlet!

@NPadrutt
Copy link
Author

I see, thanks! I will keep that in mind for some future improvements :)

Thanks again for your support and work!

@NPadrutt
Copy link
Author

Unfortunately I have to reopen this.. I actually didn't reactivate coverage in the config (stupid me..).

Can you tell me how you made that performance profile in the comment you linked?

@NPadrutt NPadrutt reopened this Mar 19, 2020
@NPadrutt
Copy link
Author

@MarcoRossignoli Alright, I was able to narrow it down a bit. Funny enough it wasn't a particular test case who was slow, but rather cover a certain project. So for me that is fixed for the moment. If you have any tipps how I could narrow that down in the future more efficently I'm very glad for every tipp :)

@MarcoRossignoli
Copy link
Collaborator

Curiosity does that test do loops?

@NPadrutt
Copy link
Author

NPadrutt commented Mar 19, 2020

It wasn't a particular test. I had to exclude the DataAccess Project for coverage. Although I do not execute the DataAccess tests, it seems that other tests would try to cover the project. Once I excluded that one the coverage is running properly (although now with the thing in issue #347 ^^)

This is my new command:
dotnet test ASK.UnitTests.sln /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:SingleHit=true /p:Exclude="[*ASKDataAccess*]*" --no-build

@MarcoRossignoli
Copy link
Collaborator

ok thanks for quick response, have a nice day and stay safe!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs more info More details are needed tenet-performance Performance related issue
Projects
None yet
Development

No branches or pull requests

2 participants