Frequent test failures in CI on Windows.Nano.1803.Amd64

[tmat]

https://dev.azure.com/dnceng/public/_build/results?buildId=237302&view=ms.vss-test-web.build-test-results-tab&runId=5884720&resultId=459291&paneView=debug

Exit code was -1073740940 but it should have been 42\r\nExpected: True\r\nActual: False

 at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose(Boolean disposing) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Microsoft.DotNet.RemoteExecutor/RemoteInvokeHandle.cs:line 143
   at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose() in /_/src/Microsoft.DotNet.RemoteExecutor/src/Microsoft.DotNet.RemoteExecutor/RemoteInvokeHandle.cs:line 55
   at System.Tests.AppDomainTests.SetShadowCopyFiles() in /_/src/System.Runtime.Extensions/tests/System/AppDomainTests.cs:line 464

[ViktorHofer]

I see more AppDomainTests randomly failing in CI, ie System.Tests.AppDomainTests.Unload

Exit code was -1073740940 but it should have been 42\r\nExpected: True\r\nActual: False

 at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose(Boolean disposing) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Microsoft.DotNet.RemoteExecutor/RemoteInvokeHandle.cs:line 143
   at System.Tests.AppDomainTests.Unload() in /_/src/System.Runtime.Extensions/tests/System/AppDomainTests.cs:line 356

https://dev.azure.com/dnceng/public/_build/results?buildId=237626&view=ms.vss-test-web.build-test-results-tab&runId=5895846&resultId=451049&paneView=debug

[ViktorHofer]

and two other failing tests: https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fpull~2F38867~2Fmerge/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190625.5/workItem/System.Runtime.Extensions.Tests

All failures happened on Nano. @MattGal this is one of the cases where a dump (on a docker instance) would be needed. Should we instead run tests manually on Nano instances somewhere?

[vcsjones]

This issue has the same exit code dotnet/corefx#38872. I ran in to similar failures over at dotnet/corefx#38845 (comment). The exit code is C0000374, which I think is usually an indicator of the process having heap corruption.

[stephentoub]

I've disabled the tests on Nano until this can be sorted out; almost every PR was failing as a result.

[ericstj]

Yes this is heap corruption. When did it start? Can we trace back to a change in CoreFx or a CoreCLR ingestion?

[ViktorHofer]

The first time it shows up in a rolling-build in master is https://dnceng.visualstudio.com/public/_build/results?buildId=237532 but that can't be the cause.

[stephentoub]

The first time it shows up

What was the nearest coreclr ingestion prior to that?

What was the first PR it showed up in?

[ericstj]

We can look in the neighborhood of that change to find culprits.

[ericstj]

dotnet/corefx#38782 is the nearest CoreCLR ingestion that picks up changes between 6/20 and 6/23. Here's the diff: https://github.com/dotnet/coreclr/compare/9bd2787a9dd2aa4d2b7d4f72afebc3dbe896e896..8974a699899bdc2cc5687504e1ada606ac803e9b

[stephentoub]

cc: @jkotas

[MattGal]

To reiterate stuff from other threads, I think this may have started (albeit less frequently) just when moving from 1709 to 1803. I got some debugging instructions from a team member and will check them out when I have time; if I get a debugging setup I will share it.

[AriNuer]

Test System.Tests.AppDomainTests/ProcessExit_Add_Remove has failed with same exit code:
Message :

Exit code was -1073740940 but it should have been 42
Expected: True
Actual:   False

Stack Trace :

   at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose(Boolean disposing) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Microsoft.DotNet.RemoteExecutor/RemoteInvokeHandle.cs:line 143
   at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose() in /_/src/Microsoft.DotNet.RemoteExecutor/src/Microsoft.DotNet.RemoteExecutor/RemoteInvokeHandle.cs:line 55
   at System.Tests.AppDomainTests.ProcessExit_Add_Remove() in /_/src/System.Runtime.Extensions/tests/System/AppDomainTests.cs:line 213

Build: -20190626.107(Master)
Failing configuration:

• Windows.Nano.1803.Amd64.Open-x64-Release
  Details:
  https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fheads~2Fmaster/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190626.107/workItem/System.Runtime.Extensions.Tests/analysis/xunit/System.Tests.AppDomainTests~2FProcessExit_Add_Remove

[ericstj]

@MattGal any update on this? We should probably re-enable Nano testing in 3.0 so keeping this in the 3.0 milestone.

[MattGal]

@ericstj I haven't had free time to try debugging in nano, but there's a good chance this week will have some spare time for me to do this for you guys.

[ericstj]

@ViktorHofer let me know that we should be able to repro this using docker for windows, running one of these simple EXEs in the container on the test shared framework: https://github.com/dotnet/corefx/blob/7094a5b94f4046abccfa2c1e6c9ab36243d8b3da/eng/pipelines/windows.yml#L110

[MattGal]

@ericstj yes I've told many people who asked about this, I'd be happy to stop by and help you do this locally as well, you just need any RS4+ machine with Docker installed. Ping me if you'd like the publicly available debugging instructions too.

[wtgodbe]

@safern I'm looking at this now

[danmoseley]

dotnet/dotnet-buildtools-prereqs-docker#157 (comment)

[safern]

@ViktorHofer put up a PR to update it 😄 dotnet/corefx#39734