Unstable test: LazyOfNonSharedPartConstructsOnlyOneInstanceAcrossThreadsV1

[AArnott]

In this Travis-CI test run we see this test failure:

[xUnit.net 00:00:32.8217164]     V1 [FAIL]
[xUnit.net 00:00:32.8228982]       System.AggregateException : One or more errors occurred. (Assert.Equal() Failure
[xUnit.net 00:00:32.8229538]       Expected: 1
[xUnit.net 00:00:32.8229830]       Actual:   0)
[xUnit.net 00:00:32.8230114]       ---- Assert.Equal() Failure
[xUnit.net 00:00:32.8230378]       Expected: 1
[xUnit.net 00:00:32.8230661]       Actual:   0
[xUnit.net 00:00:32.8241112]       Stack Trace:
[xUnit.net 00:00:32.8254893]            at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[xUnit.net 00:00:32.8255617]            at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
[xUnit.net 00:00:32.8256053]            at System.Threading.Tasks.Task.Wait()
[xUnit.net 00:00:32.8256716]         /home/travis/build/Microsoft/vs-mef/src/tests/Microsoft.VisualStudio.Composition.Tests/ThreadSafetyTests.cs(188,0): at Microsoft.VisualStudio.Composition.Tests.ThreadSafetyTests.LazyOfNonSharedPartConstructsOnlyOneInstanceAcrossThreads(IContainer container, Boolean permitMultipleInstancesOfNonSharedPart)
[xUnit.net 00:00:32.8257431]         /home/travis/build/Microsoft/vs-mef/src/tests/Microsoft.VisualStudio.Composition.Tests/ThreadSafetyTests.cs(133,0): at Microsoft.VisualStudio.Composition.Tests.ThreadSafetyTests.LazyOfNonSharedPartConstructsOnlyOneInstanceAcrossThreadsV1(IContainer container)
[xUnit.net 00:00:32.8257887]         ----- Inner Stack Trace -----
[xUnit.net 00:00:32.8258389]         /home/travis/build/Microsoft/vs-mef/src/tests/Microsoft.VisualStudio.Composition.Tests/ThreadSafetyTests.cs(182,0): at Microsoft.VisualStudio.Composition.Tests.ThreadSafetyTests.<>c__DisplayClass8_0.<<LazyOfNonSharedPartConstructsOnlyOneInstanceAcrossThreads>b__2>d.MoveNext()
Failed   V1
Error Message:
 System.AggregateException : One or more errors occurred. (Assert.Equal() Failure
Expected: 1
Actual:   0)
---- Assert.Equal() Failure
Expected: 1
Actual:   0
Stack Trace:
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Microsoft.VisualStudio.Composition.Tests.ThreadSafetyTests.LazyOfNonSharedPartConstructsOnlyOneInstanceAcrossThreads(IContainer container, Boolean permitMultipleInstancesOfNonSharedPart) in /home/travis/build/Microsoft/vs-mef/src/tests/Microsoft.VisualStudio.Composition.Tests/ThreadSafetyTests.cs:line 188
   at Microsoft.VisualStudio.Composition.Tests.ThreadSafetyTests.LazyOfNonSharedPartConstructsOnlyOneInstanceAcrossThreadsV1(IContainer container) in /home/travis/build/Microsoft/vs-mef/src/tests/Microsoft.VisualStudio.Composition.Tests/ThreadSafetyTests.cs:line 133
----- Inner Stack Trace -----
   at Microsoft.VisualStudio.Composition.Tests.ThreadSafetyTests.<>c__DisplayClass8_0.<<LazyOfNonSharedPartConstructsOnlyOneInstanceAcrossThreads>b__2>d.MoveNext() in /home/travis/build/Microsoft/vs-mef/src/tests/Microsoft.VisualStudio.Composition.Tests/ThreadSafetyTests.cs:line 182

I believe this is because both LazyOfNonSharedPartConstructsOnlyOneInstanceAcrossThreadsV1 and LazyOfNonSharedPartConstructsOnlyOneInstanceAcrossThreadsV2 reset the same static field to 0 and increment that field during the test, so if the xunit test runner runs these two methods in parallel, we have cross-contamination of the tests. We should make these tests thread-safe.

[ZoeyR]

According to this doc xUnit does not run tests in the same class in parallel. Has that behavior been overridden for these tests?

[AArnott]

Ah yes, I wasn't sure but just based on evidence the only explanation I had was that the two methods ran in parallel. Maybe the tests run one after the other but one still has async work executing after it exits?

[ZoeyR]

My guess is that it might be possible for the Assert to be hit without either of the tasks running if we're particularly unlucky.