Parallel TestContext output is not set to the correct test

[StevenDenman]

Description

When outputting text via TestContext and running the tests in parallel, the output is not set to the correct test

Steps to reproduce

Have multiple classes which output text and check the resulting trx file.
Note: Running the tests via Visual Studio does not seem to produce this, but when running the dotnet command manually
dotnet test MSTest_TestContextIssues.dll -l:trx;LogFileName=TestResults.xml

Expected behavior

The output is set to the correct test

Actual behavior

My test is simply outputting the class name and the test method and in a single test it has outputted:

        <StdOut>Std: UnitTest4 - TestMethod1&#xD;
Std: UnitTest7 - TestMethod1&#xD;
Std: UnitTest5 - TestMethod1&#xD;
Std: UnitTest8 - TestMethod1&#xD;
Std: UnitTest6 - TestMethod1&#xD;
Std: UnitTest2 - TestMethod1&#xD;
Std: UnitTest3 - TestMethod1&#xD;
Std: UnitTest1 - TestMethod1&#xD;
&#xD;


TestContext Messages:
TC: UnitTest1 - \u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000TestMethod1\u0000\u0000&#xD;
: UnitTest7 - TestMethod1&#xD;
TC: UnitTest5 - TestMethod1&#xD;
TC: UnitTest6 - TestMethod1&#xD;
TC: UnitTest8 - TestMethod1&#xD;
TC: UnitTest2 - TestMethod1&#xD;
TC: UnitTest3 - TestMethod1</StdOut>

Environment

The fix provided for v2.2.9 seems to have fixed it for std output when running the tests in Visual Studio.

But the issue still persists when running with dotnet test

Refer sample project here

[krijohan]

I have also run into this issue: console output (Console.WriteLine, no explicit use of TestContext) from tests running in parallell is mixed up when running dotnet test, but works as expected in Visual Studio.

[Evangelink]

Hey @StevenDenman @krijohan thanks for raising this issue and sorry it slipped triage. We will be investigating it soon.

[Evangelink]

@StevenDenman @krijohan I have managed to reproduce the issue. Thank you!

[Gevil]

Any update on this issue?
We seem to be getting mixed test output regardless of using Console.WriteLine or TestContext.WriteLine still.
It looks like it works fine when running tests parallel in visual studio, but the issue is there when running our tests from DevOps VsTask as well.

[Evangelink]

Hi there,

I just looked at the code to understand what's going-on and how things are setup and I understand what's the issue. While the fix is not technically super challenging it risks to impact lot of places in the codebase. I will try to confirm if I can avoid touching any public API in this case we could work on it for some 3.xx release otherwise we will have to postpone it for a 4.xx release.

Technical note: We need to replace the various stream globally and only once with a custom class. Before the call of any test, we would use AsyncLocal feature to set a unique identifier (test ID?) that would be used in the custom class to classify received (for example adding to a ConcurrentDictionary). At the end of the test execution, we can retrieve the entry and add it to the test result.

[nohwnd]

@Evangelink What you describe is what was done int 2.2.9 release, there is async local that is globally replacing Console.Out stream so when you do Console.WriteLine the output will be collected to the correct test output, no matter if it was the test writing it or any of the code it invoked (as long as task context inheritance is not disabled).

Both sync and async tests are running inside of a Task in MSTest so this works just fine, it was just a little bit difficult to get it right from what I recall.

I did not know you can write output via TestContext, so I probably did not correctly link it to use the same way of outputting, which would explain the discrepancy between Console.WriteLine and TestContext, under VS.

---

The second problem, that @krijohan mentions is that dotnet test (when used with csproj or sln) runs single vstest.console per test project, so when tests run on process level parallelism you have no way of synchronizing the output, and it ends up mixed. Their comment does not clarify if they use it with csproj/sln or list of dlls though, but you probably want to consider this when reproducing.

[krijohan]

I can reproduce mixed output with TestContext.WriteLine using dotnet test on dll:s. (But maybe the dll vs. sln/csproj thing is only relevant for Console.WriteLine?)