Running Test.cmd on a non english machine causes lots of unit tests to fail

[MaStr11]

Version Used:

Latest master at commit 69e99a8

Steps to Reproduce:

Run Test.cmd with a non english locale.

Expected Behavior:

All tests pass without errors.

Actual Behavior:

26 out of 41 test assemblies fail with errors. There were already about five failing tests in the past because of localization problems but since the introduction of xlf the situation got unbearable with hundreds of failing tests. I'm not sure but I think PR #23744 is to blame.
Attached you can find the UnitTestResults.zip
In the current situation I'm not able to run all tests locally to find regressions before committing. Maybe xUnit should be forced to use the en-US locale.

[tannergooding]

FYI. @jaredpar (compiler tests), @jinujoseph (IDE/Scripting tests)

[tannergooding]

Also FYI @jasonmalinowski

[MaStr11]

Update: Corrected the number of failures.

I did a grep 'Failures: <a href="#failures"><b>' *.* on my test results linked in the issue with this result (If I got this right, the number should show the number of failing tests):

Test	Failures	PR
Microsoft.CodeAnalysis.CSharp.Scripting.Desktop.UnitTests.dll.html	2	#24407
Microsoft.CodeAnalysis.CSharp.Scripting.UnitTests.dll.html	19	#24407
Microsoft.CodeAnalysis.VisualBasic.Scripting.UnitTests.dll.html	5	#24407
Roslyn.Compilers.CompilerServer.UnitTests.dll.html	21	#24460
Roslyn.Compilers.CSharp.CommandLine.UnitTests.dll.html	3	#24460
Roslyn.Compilers.CSharp.Emit.UnitTests.dll.1.html	2	#24460
Roslyn.Compilers.CSharp.Emit.UnitTests.dll.2.html	6	#24460
Roslyn.Compilers.CSharp.Emit.UnitTests.dll.3.html	3	#24460
Roslyn.Compilers.CSharp.Semantic.UnitTests.dll.html	4	#24539
Roslyn.Compilers.CSharp.Symbol.UnitTests.dll.html	18	#24460
Roslyn.Compilers.CSharp.Syntax.UnitTests.dll.html	2	#24539
Roslyn.Compilers.VisualBasic.CommandLine.UnitTests.dll.html	1	#24460
Roslyn.Compilers.VisualBasic.Emit.UnitTests.dll.html	1	#24539
Roslyn.Compilers.VisualBasic.Semantic.UnitTests.dll.html	8	#24539
Roslyn.Compilers.VisualBasic.Symbol.UnitTests.dll.html	1	#24539
Roslyn.Compilers.VisualBasic.Syntax.UnitTests.dll.html	5	#24539
Roslyn.ExpressionEvaluator.CSharp.ExpressionCompiler.UnitTests.dll.html	7	#24424
Roslyn.ExpressionEvaluator.VisualBasic.ExpressionCompiler.UnitTests.dll.html	2	#24424
Roslyn.InteractiveHost.UnitTests.dll.html	35	#24407
Roslyn.Services.Editor2.UnitTests.dll.html	4	#24426
Roslyn.Services.Editor.CSharp.UnitTests.dll.1.html	8	#24426
Roslyn.Services.Editor.CSharp.UnitTests.dll.2.html	1	#24426
Roslyn.Services.Editor.CSharp.UnitTests.dll.5.html	4	#24426
Roslyn.Services.Editor.VisualBasic.UnitTests.dll.1.html	2	#24426
Roslyn.Services.Editor.VisualBasic.UnitTests.dll.3.html	2	#24426
Roslyn.Services.UnitTests.dll.html	6	#24426
Total	172

[jaredpar]

Hmm, I wonder if our non-English test passes of the unit tests have caught up with our new localization process. Seems like some sort of break down has happened here. Following up internally to see what's happening.

[jmarolf]

@dotnet/roslyn-infrastructure Can we have an Jenkins run that is non-english so we don't regress this once its fixed?

[jaredpar]

@jmarolf that is my intuition on how to fix this. But i'm not sure how well it's going to play with our XLF file process.

[tannergooding]

But i'm not sure how well it's going to play with our XLF file process.

Assuming we are pulling the expected value from the resource file and the actual value is being given by the product, it should just work. (CC. @tmeschter, @nguerrera)

[jasonmalinowski]

Agreed with @tannergooding. And yes, now that we have actual loc we can actually test said loc. 😄 Any idea @jaredpar if somebody already has a non-en-US image already setup in Jenkins we could piggy back off of?

[tmeschter]

@tannergooding Yes, I expect that most of these are issues where the test is hard-coding a value that it should instead be reading from the resources.

[sharwell]

@iSeiryu That would be an unrelated issue. I'm going to collapse the comments since they are off-topic for this issue, but please feel free to create a new issue with the content.