dotnet-dump,gcdump,trace frozen and not returning

[AurelioBelletti]

Description

We are running a dotnet core 3.1 application in AKS (Linux) and we recently observed one of our application containers taken out of rotation for failing a readiness probe. The application is completely unresponsive and we wanted to collect CPU and memory traces. When we execute either of these commands dotnet-dump, dotnet-gcdump, dotnet-trace to collect information all of these commands never return and create the dumps/traces. They are stuck on creating the dumps. The container has write permissions etc and we never had problems collecting dumps/info. Also this never happened before hence we cannot reproduce this problem easily - however I have 1 POD currently on this state and if we can act quickly on this issue we might be able to capture the repro and fixes.

Configuration

.NET Core SDK (reflecting any global.json):
Version: 3.1.406
Commit: f42e6b201a

Runtime Environment:
OS Name: debian
OS Version: 10
OS Platform: Linux
RID: debian.10-x64
Base Path: /usr/share/dotnet/sdk/3.1.406/

Host (useful for support):
Version: 3.1.12
Commit: 0267ad09c6

.NET Core SDKs installed:
3.1.406 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.App 3.1.12 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.12 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

[hoyosjs]

@AurelioBelletti Sorry about this. There's one case there this could happen: if the runtime is actually "stopped" and the transport mechanisms become unresponsive as a result. If you have access to the container, right next to coreclr (libcoreclr.so) there's a dll called createdump. If the container is started with the SYS_CAP_PTRACE capability (or privileged) you can use it to collect a dump of the app.

Other things to check:

• Can you see the app if you do something like dotnet-dump ps.
• Can you try to see the permissions and ownership on files like createdump in the folder of the runtime you use. And does it match the effective user ID for the process you started?

[AurelioBelletti]

We reproed this problem again. The container has no SYS_CAP_PTRACE capabilities but this didn't prevent us in the past (and from other healthy containers) from collecting dumps. I have another POD in this state and if we don't do something quickly we might lose the repro again. None of the diagnostics tools work.

The runtime seems to be running because dotnet-gcdump ps shows the above result

[hoyosjs]

Sadly doing PS only tells us that there's a transport mechanism up and running, which is good, but it doesn't tell us much more.

Few questions and ideas:

• Had you run any diagnostic commands before against this process, or is this the first command you ran and what command was it?
• Can you try to see the permissions and ownership on files like createdump in the folder of the runtime you use. And does it match the effective user ID for the process you started? We've had cases where people were having issues to collect diagnostics due to permission issues on the createdump file, even though their app was running correctly even if the ownership of the files wasn't appropriate. This seemed to be a behavior specific to Kubernetes' runAs support.
• If you were able to collect a dump without the capability, my guess is you used dotnet-dump? That's expected behavior if so. Another way we can try to collect some data about this if possible is to use enviroment variables to capture a dump, and then send a SIGILL or similar to the process causing it to abort. The core should be saved to a path accessible in case the container dies (not sure how easy it is to do this in your type of environment).

@josalem in case you have any other ideas of what might be causing this behavior.

[AurelioBelletti]

"Had you run any diagnostic commands before against this process, or is this the first command you ran and what command was it?"

• Yes, many times recently and in the past. We have never had any problems taking any type of dumps. gcdump, full dump, traces.

"Can you try to see the permissions and ownership on files like createdump in the folder of the runtime you use. And does it match the effective user ID for the process you started? We've had cases where people were having issues to collect diagnostics due to permission issues on the createdump file, even though their app was running correctly even if the ownership of the files wasn't appropriate. This seemed to be a behavior specific to Kubernetes' runAs support."

• We are running as root. When remoting into the container we run as root. We have no problems taking dumps from other PODs of the same service/container that isn't "frozen".

"If you were able to collect a dump without the capability, my guess is you used dotnet-dump? That's expected behavior if so. Another way we can try to collect some data about this if possible is to use enviroment variables to capture a dump, and then send a SIGILL or similar to the process causing it to abort. The core should be saved to a path accessible in case the container dies (not sure how easy it is to do this in your type of environment)."

• We only use dotnet-dump, dotnet-gcdump, dotnet-trace commands. For the environment variable process, can I set the env vars and capture the dump without losing the repro? I still have 1 POD in this situation so if you want to try things let me know.

[josalem]

It sounds to me like the target process is halted and, by extension, the diagnostic server as well. This is a bad state since it means we can't use the tools to retrieve information.

Yes, many times recently and in the past. We have never had any problems taking any type of dumps. gcdump, full dump, traces.

I think what @hoyosjs meant was whether you had invoked any other commands on this specific instance of your application before it went into this state.

Does the application do anything before getting into this state, e.g., log something to the console or similar? Is there evidence that managed code ever executes?

Our next best bet is to fall back to platform tools. You can try installing LLDB and attaching to the process to inspect the stacks. Alternatively, you could use the perfcollect script to collect a native CPU trace. Either of these should give us a better idea of what's going on.

[sywhang]

perfcollect expects SYS_CAP_PTRACE capabilities so it won't work if you run it in a container without SYS_CAP_PTRACE enabled.

[AurelioBelletti]

I have added SYS_PTRACE and SYS_ADMIN capabilities to the specs of my containers. We lost the repro now but if we get a new one hopefully we can debug further... it is suspicious that the dotnet tools are hanging also and not being able to do what they are expected to do... Juan suggested that it could be related with this deadlock: https://sourceware.org/bugzilla/show_bug.cgi?id=25847

[tommcdon]

Closing this issue for now while we are waiting for a repro of the issue. Please open a new issue once the problem re-occurs.

[hoyosjs]

Closing this as the issue is happening within the runtime: dotnet/runtime#47700