dotnet-dump on Debian 9 fails with Core dump generation FAILED

[TheAngryByrd]

So I'm getting two errors in the context of running dotnet-dump on Debian 9. This is a VM and not a docker container.

The first is a permission error.

$ /opt/dotnet/tools/dotnet-dump collect -p 19668
Writing minidump with heap to /home/userfolder1/core_20190826_201258
Unhandled exception: System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (13): Permission denied /tmp/dotnet-diagnostic-19668-22628141-socket
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcClient.GetTransport(Int32 processId) in /_/src/Microsoft.Diagnostics.Tools.RuntimeClient/DiagnosticsIpc/IpcClient.cs:line 55
   at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcClient.SendMessage(Int32 processId, IpcMessage message) in /_/src/Microsoft.Diagnostics.Tools.RuntimeClient/DiagnosticsIpc/IpcClient.cs:line 67
   at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsHelpers.GenerateCoreDump(Int32 processId, String dumpName, DumpType dumpType, Boolean diagnostics) in /_/src/Microsoft.Diagnostics.Tools.RuntimeClient/DiagnosticsHelpers.cs:line 46
   at Microsoft.Diagnostics.Tools.Dump.Dumper.Collect(IConsole console, Int32 processId, String output, DumpTypeOption type) in /_/src/Tools/dotnet-dump/Dumper.cs:line 67
   at System.CommandLine.Invocation.CommandHandler.GetResultCodeAsync(Object value, InvocationContext context)
   at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass2_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Invocation.InvocationExtensions.<>c.<<UseParseErrorReporting>b__16_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Invocation.InvocationExtensions.<>c__DisplayClass8_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Invocation.InvocationExtensions.<>c.<<UseSuggestDirective>b__7_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Invocation.InvocationExtensions.<>c.<<UseParseDirective>b__6_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Invocation.InvocationExtensions.<>c.<<UseHelp>b__14_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass3_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Invocation.InvocationExtensions.<>c.<<RegisterWithDotnetSuggest>b__17_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Invocation.InvocationExtensions.<>c__DisplayClass5_0.<<UseExceptionHandler>b__0>d.MoveNext()

To which the answer is to always sudo right? but after I run that I get "Core dump generation FAILED"

$ sudo /opt/dotnet/tools/dotnet-dump collect -p 19668
Writing minidump with heap to /home/userfolder1/core_20190826_201301
Core dump generation FAILED 0x80004005

---

General info:

$/opt/dotnet/tools/dotnet-dump --version
3.0.0-preview8.19412.1+a7f67ec8e78640b0d73e123c5ee66811afb8be35

$dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview8-013656
 Commit:    8bf06ffc8d

Runtime Environment:
 OS Name:     debian
 OS Version:  9
 OS Platform: Linux
 RID:         debian.9-x64
 Base Path:   /usr/share/dotnet/sdk/3.0.100-preview8-013656/

Host (useful for support):
  Version: 3.0.0-preview8-28405-07
  Commit:  d01b2fb7bc

.NET Core SDKs installed:
  3.0.100-preview8-013656 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.0.0-preview8.19405.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.0.0-preview8-28405-07 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

[mikem8361]

You shouldn't need sudo/superuser unless the app was launched as root or another user (and maybe not even then).

Including the rest of the diagnostic team @dotnet/dotnet-diag to see if there is anything obvious.

[TheAngryByrd]

The process I'm trying to dotnet-dump is running as another user.

[hoyosjs]

In the case of a different use, this could be explained by the permissions on the pipe. After all, the pipe is created with user rights. @josalem. As for the createdump failure regarding access violation, I wonder if we have permissions to trace a different process and write them to a file. Does this repro with any app you're trying to dump as a different user?

[TheAngryByrd]

The box I was attempting to do a dump on was a production service. I'll try to repro with a smaller app testing when running as a service user vs the same user.

[TheAngryByrd]

Ok I can confirm dumping the service as the same user works but fails with Permission denied /tmp/dotnet-diagnostic-19668-22628141-socket when trying to dump a process from another user.

[TheAngryByrd]

After running:

$ sudo -H -u myserviceuser bash -c '/opt/dotnet/tools/dotnet-dump collect -p 19668'

It does bypass the Permission denied but I still fail with Core dump generation FAILED 0x80004005. So there must be something about this process, the hello world example I just tested on seemed fine.

[TheAngryByrd]

Oh hmm running

 sudo -H -u myserviceuser bash -c '/opt/dotnet/tools/dotnet-dump collect -p 14105'

On the hello world app produces the same error. Could this be a path or permission thing related to loading some files that dotnet-dump needs?

[hoyosjs]

That's a possibility, although sounds a bit hard to believe. The runtime will try to do the dumping using a binary that's expected to be side by side with the runtime. It sounds possible the path could mess this up. Also, is the runtime deployed in any special way? Is the service a self-contained app?

[TheAngryByrd]

Ok so here's a better layout of my result as to control my rambling:

My Application	dotnet-dump	Result
MyUser	MyUser	Works as expected
MyServiceUser	MyUser	Permission denied /tmp/dotnet-diagnostic-19668-22628141-socket
MyServiceUser	MyServiceUser	Core dump generation FAILED 0x80004005
MyServiceUser	sudo	Core dump generation FAILED 0x80004005

[TheAngryByrd]

We used to deploy this application as a self-contained app but have recently switched to the cross platform version and are running it with dotnet installed on the machine.

However with having the same results with a helloworld app I don't think this matters.

[TheAngryByrd]

The helloworld app for reference (in F#):

open System

[<EntryPoint>]
let main argv =
    let duration = 100
    printfn "waiting for %d seconds" duration
    Threading.Thread.Sleep(duration * 1000)
    printfn "Hello World from F#!"
    0 

[TheAngryByrd]

Oh, ok i think I found the issue with the MyServiceUser/MyServiceUser scenario. I was running the dump from my user folder and since MyServiceUser doesn't have permission to write to my user folder it was failing with that Core dump generation FAILED 0x80004005. After looking at the output from the helloworld app, it was writing:

Could not open output /home/myuser/core_20190826_214053: Permission denied

So moving to /tmp/ works just fine in the MyServiceUser/MyServiceUser.

[TheAngryByrd]

I guess this now is just more of a user experience/documentation issue.

1. Possibly give a better error than Core dump generation FAILED 0x80004005? I'm unsure if the channel dotnet-dump talks through is able to gather that?
2. Possibly default to the users temp folder instead of current? Probably a breaking change.
3. Add the FAQ what to do when you get the Permission denied /tmp/dotnet-diagnostic-19668-22628141-socket error

[mikem8361]

1. I would definitely like to make this better but there is only a HRESULT for status back from the runtime when the createdump fails.
2. You can use the dotnet-dump collect --output option to specific the file/path. Changing the default may be a breaking change like you said.
3. I'll add that to the FAQ.

[mikem8361]

I'm adding a --diag option to dotnet-dump collect so the dump creation logging is displayed. Unfortunately it is on the target process console like the above Permission denied message, but there really isn't a lot we can do for 3.x since this all happens on the runtime side. The dotnet-dump side really only gets a pass/fail indication even though it looks like an HRESULT (always E_FAIL).

[noahfalk]

there really isn't a lot we can do for 3.x since this all happens on the runtime side.

Even if we only get an HRESULT back from the runtime, we could still check known problematic conditions in advance? In the cases here I think we know what the current user id is, and we know where we are requesting to write the dump. If that user id doesn't have write access to that location we could generate a informative error message about that in dotnet-dump without having asked the runtime to do any work.

[TheAngryByrd]

Thanks to all you for getting back to me so quickly and letting me bounce ideas off you!

[hoyosjs]

@TheAngryByrd thanks for all the feedback :) it's hard to imagine all the different setups people may have and it's hard to document these scenarios if we don't hear of their existence in the wild. Hope your scenario got unblocked, and lets us know if you need any help with other bugs or behaviors you think should be documented.

[tommcdon]

Moving to 5.0 to track this portion of the issue:

1. Possibly give a better error than Core dump generation FAILED 0x80004005? I'm unsure if the channel dotnet-dump talks through is able to gather that?
2. Possibly default to the users temp folder instead of current? Probably a breaking change.

This part is done:

3. Add the FAQ what to do when you get the Permission denied /tmp/dotnet-diagnostic-19668-22628141-socket error

[mikem8361]

I created this issue about better error messaging: https://github.com/dotnet/diagnostics/issues/733.

Changing the to use the temp directory is too much of a breaking change.

[mikem8361]

Hopefully I have addressed your concerns.