Performance degradation on many tests

[timcassell]

I've been noticing performance degradation when running many tests (thousands in my project) for a while. It's especially noticeable on my long-running tests, but the test times also show degradation on quick tests also.

I have setup and teardowns for every test.

At first I thought it was because I have objects pooling and the pool could be causing a leak and interfering with the performance, but I still see the same degradation with the pool disabled.

When I run all tests in a group, the last sub-group takes 1.4s. Then when I run just the last sub-group independently, it takes 480ms, nearly 1/3 of the time. The more tests I run at a time, the longer it takes the last test to complete. This is a big problem for me with over 3600 tests, half of them are long-running (the images are only showing the quick tests).

I was using NUnit 3.13.0, and I just updated to NUnit 3.13.2 and saw the same behavior on both versions.

Any ideas?

[Edit] I should add that my library is targeting netstandard2.0, and I'm running the tests in net472 and net5.0 (I see the same degradation in both runtimes).

[mikkelbu]

@timcassell Do you see the same degradation when you run the tests using the NUnit console - https://github.com/nunit/nunit-console/releases/tag/v3.12 - and/or NUnitLite - see e.g. https://docs.nunit.org/articles/nunit/running-tests/NUnitLite-Runner.html

When you are running the tests in VS you are also depending on VS, VSTest, and the NUnit 3 Test Adapter, so the above suggestion is to determine where the problem is. Note that both the NUnit console and the NUnit 3 Test Adapter uses the NUnit engine.

[timcassell]

@mikkelbu Interestingly, I got the opposite effect with the nunit3-console. Though I could only run the net472 tests with it, not net5.0.

nunit3-console ProtoPromiseTests.dll --where "namespace == ProtoPromiseTests.APIs"

SequenceTests duration="0.539729"

nunit3-console ProtoPromiseTests.dll --where "class = ProtoPromiseTests.APIs.SequenceTests"

SequenceTests duration="0.606604"

And I'm not sure if the nunit3-console executes the tests in the same order as the VS test runner (though I'm guessing it does if the output XML is any indication).

[lahma]

When someone says thousands of tests this might be the same problem I encountered with my large parametrized test suite of 86k tests, more information in the PR here #4034 . You examples use a specific filter so it shouldn't be a filtering issue, but it might be the load phase which I also tried to address.

[rprouse]

I believe this is fixed with #4034, please reopen if I am wrong.

[OsirisTerje]

@lahma Is this also fixed in main ? Do you have the PR for that?

[stevenaw]

@OsirisTerje
I think I ported this in #4053
I may've set the milestone incorrectly there. On mobile at the moment and can't check too in-depth

[OsirisTerje]

@stevenaw Thanks! That's cool! #4053 is in and correct :-)

[lahma]

Yes I can also confirm that the linked PR brought the intended changes to main too 👍🏻