Add dump creation support to RemoteExecutor for Linux

[danmoseley]

Sometimes tests hang. When they do, it's very often necessary to have a dump to figure out the problem. When they hang in RemoteExecutor, thanks to #4085 we should get a nice dump on Windows.

We should make this work on Linux as well, presumably using dotnet dump (https://github.com/dotnet/diagnostics/blob/master/documentation/dotnet-dump-instructions.md)

@epananth this would require it be installed on the Linux test machines using dotnet tool install -g dotnet-dump. How do you suggest we make that happen? Should remote executor itself invoke that?

cc @stephentoub

[epananth]

@danmosemsft will need to get back to you on this. I learned from @ChadNedzlek that dotnet might not work on some linux. So this might mess up the machines. So this is kind of tricky.

[markwilkie]

@mikem8361 - does the dump tool work on all core supported platforms?

[danmoseley]

@markwilkie according to the link above it sounds like it works on all platforms/architectures except macOS (where we have never had a way to make dumps, as far as I know)

[mikem8361]

Yes, dotnet-dump collect should work on all architectures/platforms except macOS. It has the runtime launch createdump to do the generation. It will only work with version 3.0 and greater of the runtime.

[markwilkie]

@epananth - let's chat about the right way to make this happen.

[ViktorHofer]

We just discussed this in Teams. We should aim to capture dumps for both hangs and crashes inside RemoteExecutor with dotnet dump (cross-plat solution). Currently we only capture dumps for hangs on Windows.

[ViktorHofer]

Additionally the dump should be created in all cases, CI and locally. On CI we want to move the dump to a well known location so that helix can pick it up:

arcade/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs

Line 146 in 3bbaf86

string uploadPath = Environment.GetEnvironmentVariable("HELIX_WORKITEM_UPLOAD_ROOT");

[michellemcdaniel]

Closing because the parent Epic was closed. If you believe this issue should still be worked on, please re-open.