Description
Same as #2821, but for the ExecutionContext rather than the SynchronizationContext. (.NET tracks them individually.)
One test should not be able to affect another test simply because the test execution scheduler happens to run them on the same worker thread. This has unstable results depending on how many worker threads there are and on how NUnit decides to order them over time. But more importantly, it's an invisible factor that influences every test so that you may not be actually testing what you think you are testing.
Each test is inheriting an unrelated ExecutionContext from the last test the worker happened to pick up. This can cause some tests to fail when they should pass. Much worse, as in some cases just now for me, it can silently render some tests ineffective (passing whether or not they should fail).
Repro
Some tests designed to demonstrate the problem by failing:
public static class SystemUnderTest
{
private static readonly AsyncLocal<int> ValueFlowedWithExecutionContext = new AsyncLocal<int>();
public static int Increment() => ++ValueFlowedWithExecutionContext.Value;
}
public static class Tests
{
// Only one test succeeds because the .NET ExecutionContext is flowed between tests
// in whatever order they happen to execute.
[Test]
public static void Test1()
{
Assert.That(SystemUnderTest.Increment(), Is.EqualTo(1));
}
[Test]
public static void Test2()
{
Assert.That(SystemUnderTest.Increment(), Is.EqualTo(1));
}
[Test]
public static void Test3()
{
Assert.That(SystemUnderTest.Increment(), Is.EqualTo(1));
}
}Workarounds
The easiest workaround I'm aware of is to put [RequireThread] on every test method. This comes with a performance hit. It's also easy to forget if it breaks your test by causing it to pass rather than by causing it to fail.
A more efficient workaround would be:
private static void DoInIsolatedExecutionContext(Action action)
{
using (var copy = ExecutionContext.Capture().CreateCopy())
{
ExecutionContext.Run(copy, state => ((Action)state).Invoke(), state: action);
}
}
[Test]
public static void TestN()
{
DoInIsolatedExecutionContext(() =>
{
Assert.That(SystemUnderTest.Increment(), Is.EqualTo(1));
});
}Concerns
Would some NUnit users be making use of this behavior as a feature when all tests execute on a single thread and they are constrained to execute in order?
public static class Tests
{
[Test, Order(1)]
public static void Test1()
{
Assert.That(SystemUnderTest.Increment(), Is.EqualTo(1));
}
[Test, Order(2)]
public static void Test2()
{
// Build on the previous test (where ‘previous’ is codified via OrderAttribute)
Assert.That(SystemUnderTest.Increment(), Is.EqualTo(2));
}
[Test, Order(3)]
public static void Test3()
{
// Build on the previous test (where ‘previous’ is codified via OrderAttribute)
Assert.That(SystemUnderTest.Increment(), Is.EqualTo(3));
}
}The SynchronizationContext is no longer flowed between tests as of 3.11 (#2821) which is very similar though it affects different things. No one has reported noticing the difference.