OneTimeTearDown runs on a new thread with mismatched Thread Name and Worker Id

[EraserKing]

I'm not sure if I'm relying on some behavior which is not guaranteed, so I'm not 100% sure if this is an issue.

I would like to have parallel testing on a class to speed up.
To make sure each worker initialize a separate instance of the specific class, I created a wrapper which returns the instance of the class under the current thread (based on Thread.CurrentThread.Name), or initialize a new instance if not initialized yet - so in the test class I would just call the wrapper without considering which thread I'm now on and which instance is the one I should get among a set of instances. It it something like:

public static class TestObjectWrapper
{
    public static TestObject Get()
    {
        if (!testObjects.ContainsKey(Thread.CurrentThread.Name))
        {
            testObjects.AddOrUpdate(Thread.CurrentThread.Name, new TestObject(), (o, n) => new TestObject());
        }
        return testObjects[Thread.CurrentThread.Name];
    }

    private static ConcurrentDictionary<string, TestObject> testObjects = new ConcurrentDictionary<string, TestObject>();
}

So no matter where I am, I only need to call TestObjectWrapper.Get(), I can get an instance.

The reason why I rely on the thread name is because the wrapper is inside another project without NUnit reference, and it's also used in some other places without NUnit (so I cannot take TestContext.CurrentContext.WorkerId instead). Altogether, I read from stackoverflow that the thread name is identical to the worker id, which is something like ParallelWorker#8, or NonParallelWorker, or null (under Debug mode).

Under most scenarios it works quite well, when I only enable test fixture level parallelism ([Parallelizable(ParallelScope.Fixtures)]), but I found an unexpected behavior for OneTimeTearDown method in the test fixtures.
In the OneTimeTearDown, I'm hoping to get the class instance to do some clean up. So I have some code like:

[Parallelizable(ParallelScope.Fixtures)]
public class TestClass
{
        // Actual test code omitted
        [OneTimeTearDown]
        public void OneTimeTearDown()
        {
            TestObjectWrapper.Get().Shutdown();
        }
}

Obviously I'm expecting getting the same instance used by test methods in this test fixture, but later I found there's high chance that this method is exectued on a completely new thread, which caused the original instance not terminated but creates a new one instead.

And during my observation I found this only happens to OneTimeTearDown methods. The other types (e.g. OneTimeSetUp, SetUp, TearDown) are still always running in the same thread of the test methods.

Then I did some research - I output TestContext.CurrentContext.WorkerId and Thread.CurrentThread.Name, and found mostly they are identical, but sometimes in OneTimeTearDown, they are not the same. TestContext.CurrentContext.WorkerId is consistent among all the methods in the test fixture, and Thread.CurrentThread.Name is the same as WorkerId, unless in the OneTimeTearDown method.

An example of the output:

Consturctor Worker Id = ParallelWorker#7
Consturctor Thread Name = ParallelWorker#7
OneTimeSetUp Worker Id = ParallelWorker#7
OneTimeSetUp Thread Name = ParallelWorker#7
SetUp Worker Id = ParallelWorker#7
SetUp Thread Name = ParallelWorker#7
TestMethod1 Worker Id = ParallelWorker#7
TestMethod1 Thread Name = ParallelWorker#7
TearDown Worker Id = ParallelWorker#7
TearDown Thread Name = ParallelWorker#7
SetUp Worker Id = ParallelWorker#7
SetUp Thread Name = ParallelWorker#7
TestMethod2 Worker Id = ParallelWorker#7
TestMethod2 Thread Name = ParallelWorker#7
TearDown Worker Id = ParallelWorker#7
TearDown Thread Name = ParallelWorker#7
OneTimeTearDown Worker Id = ParallelWorker#7
OneTimeTearDown Thread Name = ParallelWorker#4 // Worker is the same, but thread is new, and name differs form Worker Id

This explains why I get a new instance other than the original one in the OneTimeTearDown method.

I thought I could have a workaround that assign the thread name from the worker id, so it can pretend it's still in the original thread and then then wrapper can return the expected instance, but later when I try to write some code like this:

       [OneTimeTearDown]
        public void OneTimeTearDown()
        {
            Thread.CurrentThread.Name = TestContext.CurrentContext.WorkerId; // Added this line
            TestObjectWrapper.Get().Shutdown();
        }

Then I found the code execution just terminates at the added line. The original shutdown line is not performed at all, and there's error message telling that - everything looks correct unless the code is not executed.
I read the documents, and tried with some attributes, like [SingleThreaded], or [RequiresThread], but they don't works as my thought - still the same behavior on the OneTimeTearDown method.

So, as a quick recap, my questions are listed below:

1. Is the consistency between thread name and worker id, an officially supported behavior?
2. Is there any way which can really make sure all the code in a test fixture (with [Parallelizable(ParallelScope.Fixtures)]) executed in the same thread & worker?
3. If 2 is not possible, if there is any way I can reassign the value of Thread.CurrentThread.Name?

Environments:
Visual Studio 2019 Enterprise
NUnit 3.13.1
NUnit3TestAdaptor 3.17.0
Test project is targeting .NET Core 3.1
Running test from VS Test Explorer of Run mode (Debug mode seems ignoring parallism, which is fine)

Thanks.

[rprouse]

We do not guarantee that the all code run within a fixture will be on the same worker thread. There are so many possible combinations of parallelism and nested fixtures that it would be very difficult for us to support.

Can you switch your code to use SetUp/Teardown instead?

You might also be able to use runsettings to limit the number of workers to one, but you will lose parallelism.

[EraserKing]

Thanks. The reason why I prefer OneTimeSetUp / OneTimeTearDown is because the starting up / cleaning up is quite time-consuming and I would like not to do that for every single test case.
Doing that in TearDown is possible, but I'm wondering if there's any way to know this is the last TearDown in the test fixture so I could do the clean up at that time?

[EraserKing]

I did some further research today.

About the question 3, the reason why the code of reassigning value fails it that the Name property can only be assigned once. So if it has been assigned before (i.e. the value is not null) and you'd like to give it a new value, it would throw an exception - but the exception in OneTimeTearDown would not show up in the report, and thus it looks like the code is not executed at all. Most of the time, the thread name is identical to the worker id but sometimes in OneTimeTearDown it would not. So it returns to the question 1, is the consistenty between the two values guaranteed?

About the question 2, I read the docs - and found this attribute: SingleThreaded, which says:

SingleThreadedAttribute is used on a TestFixture and indicates that the OneTimeSetUp, OneTimeTearDown and all the child tests must run on the same thread.
When using this attribute, any ParallelScope setting is ignored.

But, as mentioned in the original issue, even if I have this attribute, OneTimeTearDown may get executed in another thread.
And, since it also mentions ParallelScope, I think it's designed to work with Parallelizable attribute.

I guess the expected behavior should be, if the class has both Parallelizable and SingleThreaded attribute, all the code in this test fixture should be exuected in the same single thread always, while the test class may be executed with other test fixtures in parallel.
But from what I see, the worker is always the same one, but the thread may be a new one when executing OneTimeTearDown.

So, considering this, I'm wondering if this is still an issue (at least the description on the doc page is not working as it says).

[rprouse]

I am going to switch this to confirm to double check that the SingleThreaded attribute isn't working as advertised. Good find by the way, it just goes to show you that even the maintainers forget all the features of NUnit! 😄

[EraserKing]

Thanks!

Just now I did some quick tests, and it looks like the attribute is indeed not working as described.

I created four test fixtures (just repeat the code below, with different class name, but exactly the same code for methods), each of them consists of two very simple test, plus additional set up / tear down / one time set up / one time tear down methods:

    [TestFixture]
    [Parallelizable]
    [SingleThreaded]
    public class Test01
    {
        [SetUp]
        public void SetUp()
        {
            TemporaryLogger.WriteLine($"Class = {this.GetType().Name}, Method = SetUp, Worker = {TestContext.CurrentContext.WorkerId}, Thread Id = {Thread.CurrentThread.ManagedThreadId}, Thread Name = {Thread.CurrentThread.Name}");
        }

        [TearDown]
        public void TearDown()
        {
            TemporaryLogger.WriteLine($"Class = {this.GetType().Name}, Method = TearDown, Worker = {TestContext.CurrentContext.WorkerId}, Thread Id = {Thread.CurrentThread.ManagedThreadId}, Thread Name = {Thread.CurrentThread.Name}");
        }

        [OneTimeSetUp]
        public void OneTimeSetUp()
        {
            TemporaryLogger.WriteLine($"Class = {this.GetType().Name}, Method = OneTimeSetUp, Worker = {TestContext.CurrentContext.WorkerId}, Thread Id = {Thread.CurrentThread.ManagedThreadId}, Thread Name = {Thread.CurrentThread.Name}");

        }

        [OneTimeTearDown]
        public void OneTimeTearDown()
        {
            TemporaryLogger.WriteLine($"Class = {this.GetType().Name}, Method = OneTimeTearDown, Worker = {TestContext.CurrentContext.WorkerId}, Thread Id = {Thread.CurrentThread.ManagedThreadId}, Thread Name = {Thread.CurrentThread.Name}");

        }

        [Test]
        public void Test1()
        {
            TemporaryLogger.WriteLine($"Class = {this.GetType().Name}, Method = Test1, Worker = {TestContext.CurrentContext.WorkerId}, Thread Id = {Thread.CurrentThread.ManagedThreadId}, Thread Name = {Thread.CurrentThread.Name}");
            Assert.Pass();
        }

        [Test]
        public void Test2()
        {
            TemporaryLogger.WriteLine($"Class = {this.GetType().Name}, Method = Test2, Worker = {TestContext.CurrentContext.WorkerId}, Thread Id = {Thread.CurrentThread.ManagedThreadId}, Thread Name = {Thread.CurrentThread.Name}");
            Assert.Pass();
        }
    }

TemporaryLogger is a very simple class, just write text to disk for tracking:

    public static class TemporaryLogger
    {
        private static object loggerLock = new object();

        public static void WriteLine(string value)
        {
            lock (loggerLock)
            {
                using FileStream fs = new FileStream(@"C:\temp\templogger.txt", FileMode.Append);
                using StreamWriter sw = new StreamWriter(fs, Encoding.UTF8);
                sw.WriteLine(value);
            }
        }
    }

The templogger.txt shows (converted into table for read, and sorted by class name):

Class	Method	Worker	Thread Id	Thread Name
Test01	OneTimeSetUp	ParallelWorker#14	27	ParallelWorker#14
Test01	SetUp	ParallelWorker#14	27	ParallelWorker#14
Test01	Test1	ParallelWorker#14	27	ParallelWorker#14
Test01	TearDown	ParallelWorker#14	27	ParallelWorker#14
Test01	SetUp	ParallelWorker#14	27	ParallelWorker#14
Test01	Test2	ParallelWorker#14	27	ParallelWorker#14
Test01	TearDown	ParallelWorker#14	27	ParallelWorker#14
Test01	OneTimeTearDown	ParallelWorker#14	20	ParallelWorker#7
Test02	OneTimeSetUp	ParallelWorker#15	28	ParallelWorker#15
Test02	SetUp	ParallelWorker#15	28	ParallelWorker#15
Test02	Test1	ParallelWorker#15	28	ParallelWorker#15
Test02	TearDown	ParallelWorker#15	28	ParallelWorker#15
Test02	SetUp	ParallelWorker#15	28	ParallelWorker#15
Test02	Test2	ParallelWorker#15	28	ParallelWorker#15
Test02	TearDown	ParallelWorker#15	28	ParallelWorker#15
Test02	OneTimeTearDown	ParallelWorker#15	24	ParallelWorker#11
Test03	OneTimeSetUp	ParallelWorker#16	29	ParallelWorker#16
Test03	SetUp	ParallelWorker#16	29	ParallelWorker#16
Test03	Test1	ParallelWorker#16	29	ParallelWorker#16
Test03	TearDown	ParallelWorker#16	29	ParallelWorker#16
Test03	SetUp	ParallelWorker#16	29	ParallelWorker#16
Test03	Test2	ParallelWorker#16	29	ParallelWorker#16
Test03	TearDown	ParallelWorker#16	29	ParallelWorker#16
Test03	OneTimeTearDown	ParallelWorker#16	28	ParallelWorker#15
Test04	OneTimeSetUp	ParallelWorker#2	15	ParallelWorker#2
Test04	SetUp	ParallelWorker#2	15	ParallelWorker#2
Test04	Test1	ParallelWorker#2	15	ParallelWorker#2
Test04	TearDown	ParallelWorker#2	15	ParallelWorker#2
Test04	SetUp	ParallelWorker#2	15	ParallelWorker#2
Test04	Test2	ParallelWorker#2	15	ParallelWorker#2
Test04	TearDown	ParallelWorker#2	15	ParallelWorker#2
Test04	OneTimeTearDown	ParallelWorker#2	23	ParallelWorker#10

It looks like all the 4 OneTimeTearDown methods are executed in the same worker of the other methods in the class, but with a new thread.

[quldude]

@rprouse do you know when we can get the fix for this?

[edwickensrhenus]

@EraserKing Just wanted to say your debugging code really helped me get some tests to run properly in parallel.
For anyone else who stumbles across this: This was the critical piece of information I was missing:
https://docs.nunit.org/articles/nunit/writing-tests/attributes/fixturelifecycle.html

[Parallelizable(ParallelScope.All)]
[FixtureLifeCycle(LifeCycle.InstancePerTestCase)]
public class MyTestClass
...
[OneTimeTearDown]
    protected static void OneTimeTearDown()
    {
        SharedContext.TearDown(); // this is a static class with shared state
    }