Execute OneTimeTearDown as early as possible when running fixtures in parallel #2335
Comments
|
You have used an assembly-level attribute to indicate that all fixtures may run in parallel. That's what they are doing. If you really don't want test1 to run in parallel with the others, you can use an attribute to indicate that. Perhaps you are asking for NUnit to somehow prioritize running teardowns. That's a feature request. I even considered doing it in the implementation but it was a fair bit more complex so I postponed it. |
|
I do want all the fixtures to run in parallel. Is OneTimeTearDown not supposed to be run at the end of the fixture it is in (like, after tests, not after tests from other fixtures)? In the result above from 3.7.1, OneTimeTearDown's are 'queued' at the end, when the worker has finished all other tests. The "Logs from NUnit 3.6.1" is what I expected to happen. |
|
OneTimeTearDown for a fixture is guaranteed to run...
That's all we guarantee currently. More is a new feature, which means you or someone else would need to scope it out fairly precisely for us to evaluate. |
|
OK, thanks for the explanation, I'll see what I can do about that feature request. |
|
You can just post here and we'll make this a feature request. One possibility is to simply give teardowns some sort of priority, without guaranteeing a particular order. That would allow resources to be released more quickly but is a fairly big change. Another is to somehow determine whether the teardown can be run on the same thread as the last test completed. That would allow faster release in many most cases, while still using normal scheduling when the teardown called for a different thread. This probably boils down to checking the required Apartment, but it would take some analysis to be sure. Note that this issue only arises if both the fixture and the test cases are run in parallel. |
|
+1 on this issue. My group is also running concurrent fixtures that can run as high as 60 per project/assembly. Each test class inherits from a base class containing OneTimeSetUp and OneTimeTearDown which manage the driver resources for that fixture. Our test runner hosts (8 x 2.9Ghz + 15GB RAM) are grinding to a halt due to memory saturation at about the 20th fixture (ie 20 open browsers, many of which are inactive). When running a smaller set of fixtures it is clear that OneTimeTearDown not executing until all fixtures have completed their test runs as drivers/browsers are not disposed until the end of the entire run. As the other reporter noted, once we rolled back our test runner pool to 3.6.1, the issue disappeared. With this version, the active driver/browser count is almost always equivalent to the configured number of workers so memory management is not a problem. I'd be happy to provide more detailed debug info if needed. p.s. Thank you for all your great work on this project over the years! |
|
Unfortunately, the behavior in 3.6.1 was incorrect. The teardown always ran on the same thread as the last test to complete. This is not a problem if the thread requirements of each test match those of the fixture, but it is incorrect if any test has different requirements (e.g. ApartmentState) and that test happens to be the last one to run. We don't need +1s on this issue, we have accepted that it's a problem. A specific proposal that takes into account the various factors would speed it along. |
|
I might have found a workaround, or it's incorrect behaviour just like in 3.6.1, as @CharliePoole stated in his previous comment. In my test setup (from my first post) I added a new base class and gave it Current test setup for NUnit 3.7.1 looks like this: using System.Threading;
using NLog;
using NUnit.Framework;
namespace Skanska.Korab2.UITests.Infrastructure.Tests
{
[Parallelizable(ParallelScope.Default)]
class Base
{
}
class Common
{
[...]
}
class Class1Fast : Base
{
[...]
}
class Class2Slow : Base
{
[...]
}
class Class3Slow : Base
{
[...]
}
class Class4 : Base
{
[...]
}
class Class5 : Base
{
[...]
}
class Class6 : Base
{
[...]
}
class Class7 : Base
{
[...]
}
class Class8Slow : Base
{
[...]
}
}AssemblyInfo remained unchanged from that in my first post (still has Any idea why |
|
I stated earlier that the problem only arises when both fixtures and tests are parallel. I'm not 100% sure of this - it may be a problem whenever the fixtures are run in parallel. Whoever works the issue will have to debug to see if that's so. But in any case, when you use |
That's what I thought, but they do run in parallel - with both
I guess in this case |
|
Interesting. That calls for some debugging I guess. FWIW, Writing the above, it begins to make sense. The attribute tricks NUnit into believing that no attribute was used, so the assembly-level attribute rules. |
|
@nunit/framework-team @nunit/core-team |
I definitely agree. This would bring a return of the memory issues we were facing around 3.0. I don't believe we've upgraded all of our test suites to 3.7 yet, but in some places this could ground things to a halt. |
|
(Of course, this is more specific, as it only affects tests with OneTimeTearDown's rather than all tests - but would otherwise cause a major issue for us.) |
|
Actually I think it may affect all tests. Even if there is no one time teardown per se, that's where we release the reference. |
CharliePoole
changed the title
|
I changed the title to reflect the objective. Since this pertains to an area I worked in, I'm assigning it to myself. |
NUnit 3.7.1
Setup:
Running fixtures in parallel (
Parallelizable(ParallelScope.Fixtures)set inAssemblyInfo.cs) in at least 2 threads (LevelOfParallelism(2)set inAssemblyInfo.cs).Multiple fixtures with tests and OneTimeTearDown's for each of them exist.
Problem:
All
[OneTimeTearDown]methods are executed in bulk when first available thread finishes.Example:
Tests:
AssemblyInfo.cs
Result:
Expected result:
Test1TearDownis run before Worker#2 has started tests from other fixtures.Previous versions:
Logs from NUnit 3.6.1:
The text was updated successfully, but these errors were encountered: