CscBench test failing spuriously

[RussKeldorph]

Test case tests\src\JIT\Performance\CodeQuality\Roslyn\CscBench.cs seems to be failing in the CI spuriously. I think I remember seeing this on both Windows and Linux, but the only examples I can find are on Ubuntu, so perhaps assume it is Linux-specific for now.

From Andy:
I wonder if it’s some kind of race condition or similar setting up the overlay test directory. I noticed in the past that it was easy to end up with different resolved versions in the various project.json files and when the overlay processing goes to flatten all these into one directory, I wonder if sometimes (or just on some machines, or just on Ubuntu, etc) it ends up with a different set of implementation assemblies.

It feels like either we should head towards having all tests in CoreCLR share a single project.json so we can’t have version mismatches like this, or have one overlay directory per project.json, or may be do something else to keep things straight.

Examples of (probably) spurious failures:
http://dotnet-ci.cloudapp.net/job/dotnet_coreclr/job/release_ubuntu_tst_prtest/969/testReport/junit/JIT.Performance.CodeQuality.Roslyn/CscBench/CscBench/
http://dotnet-ci.cloudapp.net/job/dotnet_coreclr/job/release_ubuntu_tst_prtest/971/testReport/junit/JIT.Performance.CodeQuality.Roslyn/CscBench/CscBench/

Sample output from CI:
MESSAGE:
BEGIN EXECUTION /mnt/j/workspace/dotnet_coreclr/release_ubuntu_tst_prtest/bin/tests/Windows_NT.x64.Release/Tests/coreoverlay/corerun CscBench.exe libust[34441/34441]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305) Unhandled Exception: System.IO.FileLoadException: Could not load file or assembly 'System.Runtime, Version=4.0.21.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040) at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.AddSyntaxTrees(IEnumerable1 trees) at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.Create(String assemblyName, CSharpCompilationOptions options, IEnumerable1 syntaxTrees, IEnumerable`1 references, CSharpCompilation previousSubmission, Type returnType, Type hostObjectType, Boolean isSubmission) at CscBench.CompileBench() at CscBench.Main() ./CscBench.sh: line 55: 34441 Aborted (core dumped) $_DebuggerFullPath "$CORE_ROOT/corerun" CscBench.exe $CLRTestExecutionArguments $Host_Args Expected: 100 Actual: 134 END EXECUTION - FAILED
+++++++++++++++++++
STACK TRACE:

[AndyAyersMS]

Failures mostly (exclusively?) happen on dci-ub-fbld-2.

[ellismg]

We hit similar things a while back in CoreFX where we constructed a test overlay by doing complicated find magic. I found that the order of enumeration of the FS would differ across machines and so we'd actually end up copying different files due to our last one wins strategy.

[AndyAyersMS]

Was curious why the closely related managed\Compilation test didn't hit this issue. Turns out it's excluded in the "unsupported outside windows list".

I hate to just disable this test too, but it is causing enough spurious failures that it seems that is the sensible thing to do.

[RussKeldorph]

@wtgodbe Can you take this? We either need to fix runtest.sh or start enforcing a compatible pattern that tests must follow (such as a single common version of each assembly).

[wtgodbe]

@RussKeldorph I can take a crack, but I may need a bit of help. @ellismg could you elaborate a bit on what you meant here?

The problem with dotnet/coreclr#2728 is due to how you build out your test runtime folders. As @AndyAyersMS on certain machines we enumerate directories differently and can end up copying over different versions of System.Runtime.dll which can cause bind errors if an older version is copied instead of a newer one.

@RussKeldorph Is there someone from your team who owns the end-to-end testing process? You should probably move away from the find magic here https://github.com/dotnet/coreclr/blob/master/tests/runtest.sh#L334. One option would be to restore a specific project.json file with the newest versions of your dependencies and copy the output of that instead to the overlay (we do something similar in CoreFX with the "test-runtime"). If there's someone that better understands exactly what you need to do for setup they can probably fix this faster than me.

[sergiy-k]

I think the wrong version of System.Runtime gets copied from the Linux.AnyCPU.Debug\RemoteExecutorConsoleApp folder.
@wtgodbe, excluding this folder from the search path can be used as short term workaround until you implement a proper fix.

[wtgodbe]

dotnet/coreclr#3241 here's a workaround as suggested by @sergiy-k, for now.