dotnet-dump in container doesn't work due to ptrace restriction

[msallin]

Hi

We run our containers on AWS EKS, and the platform has forbidden calls to "ptrace".
The image we use is based on the newest version of mcr.microsoft.com/dotnet/aspnet:7.0-alpine.
We made minor adjustments, like installing dotnet-dump beside the application to perform memory dumps in production.

When we run dotnet-dump we get an error.

/tmp $ /diag/dotnet-dump ps
1  xyxyxy

/tmp $ /diag/dotnet-dump collect --process-id 1

Writing full to /tmp/core_20230120_095245
[createdump] Problem suspending threads: ptrace(ATTACH, 1) FAILED Operation not permitted (1)
[createdump] Failure took 0ms

/tmp $

As mentioned in dotnet/docs#20573 (comment) from @shirhatti this should work because the tool should send a signal to the process to dump itself.

sh-4.2$ sysctl kernel.yama.ptrace_scope
kernel.yama.ptrace_scope = 0

However, we see that the collect tries to do a ptrace ATTACH.

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Hi

We run our containers on AWS EKS, and the platform has forbidden calls to "ptrace".
The image we use is based on the newest version of mcr.microsoft.com/dotnet/aspnet:7.0.
We made minor adjustments, like installing dotnet-dump beside the application to perform memory dumps in production.

When we run dotnet-dump we get an error.

/tmp $ /diag/dotnet-dump ps
1  xyxyxy

/tmp $ /diag/dotnet-dump collect --process-id 1

Writing full to /tmp/core_20230120_095245
[createdump] Problem suspending threads: ptrace(ATTACH, 1) FAILED Operation not permitted (1)
[createdump] Failure took 0ms

/tmp $

As mentioned in dotnet/docs#20573 (comment) from @shirhatti this should work because the tool should send a signal to the process to dump itself.

sh-4.2$ sysctl kernel.yama.ptrace_scope
kernel.yama.ptrace_scope = 0

However, we see that the collect tries to do a ptrace ATTACH.

Author:	msallin
Assignees:	-
Labels:	area-Diagnostics-coreclr, untriaged
Milestone:	-

[mkilchhofer]

As I am one of the Kubernetes Platform Engineers at @swisspost who is responsible for @msallin 's issue I wanted to add some platform specific stuff:

• We are on Kubernetes 1.22 today
• We use "Amazon Linux 2" nodes with kernel 5.4.219-126.411.amzn2.x86_64
• Container runtime is containerd 1.6.6
• Currently we are still on PSP (Pod Security Policy) and enforce our workloads to drop ALL caps
• Recommend that users run with seccomp RuntimeDefault:

        securityContext:
          # ..
          seccompProfile:
            type: RuntimeDefault

• Sysctl regarding YAMA as @msallin mentioned

If we remove the seccompProfile configuration or extend the securityContext with

          securityContext:
            # ..
            capabilities:
              add:
                - SYS_PTRACE
              drop:
                - ALL

Then it works.

But allowing every dotnet workload in our Kubernetes clusters to run with SYS_PTRACE or no seccompProfile is not an option as it opens an unneccessary attack vector:

• https://www.hackitude.in/docker-security/container-breakouts/abusing-capabilities/sys_ptrace
• https://blog.aquasec.com/aqua-3.2-preventing-container-breakouts-with-dynamic-system-call-profiling

xref:

• Permission for dotnet-dump in Docker docs#20573 (comment)

[tommcdon]

Hi @mkilchhofer! Adding @mikem8361 @hoyosjs. Thank you for the details! As you have correctly found, ptrace is a known dependency for .NET dump functionality on Linux. Createdump uses ptrace to suspend and inspect thread state in the target process, and therefore depends on the SYS_PTRACE capability. We would be open to discussion and community contributions if you have thoughts on how we could work around this limitation.

[hoyosjs]

@msallin, The comment in the docs is saying:

The forked process now connects back to it’s parent process via PTRACE and is able to dump the process’ virtual memory.

Our official docs FAQ also mentions this fact:

https://learn.microsoft.com/en-us/dotnet/core/diagnostics/faq-dumps#why-can-t-i-collect-dumps-when-running-inside-a-container-

The signaling mechanism is so that on-demand dumps can be collected without requiring elevation. Adding the CAP_SYS_PTRACE shouldn't be needed in containerd and docker backed containers using the default profile: https://github.com/containerd/containerd/blob/3f565daf682f652a13bbafeff63bc618377e1b71/contrib/seccomp/seccomp_default.go#L494. However, it looks like that change is slated for containerd 1.7. Moby has had such profile working for a while. However, once that change goes in it is highly recommendable to set ptrace_settings in yama to 1 such that only elevated processes can trace arbitrary processes, and otherwise you can only trace child processes or parent processes that opt into it.

[mkilchhofer]

@hoyosjs
Thanks for the quick answer and the technical explanation.

One thing I cannot follow:
How comes the YAMA feature into play? The main process from our customer (@msallin) runs as PID 1 inside the container. In case where a memory dump is needed, they exec into the Pod and run the dotnet-dump command. This command is neigher in a parent/child relationship to the process that needs to be dumped. It's a sibling process and therefore PTRACE is not allowed in my opinion.

[noahfalk]

The way dotnet-dump works is that it sends a message over a unix domain socket to the .NET process. Upon receiving that message, the .NET process forks and runs the 'createdump' process. createdump is the one that invokes PTRACE on its parent, the .NET process.

[mkilchhofer]

Okay, that makes sense.

I went through the containerd releases and found out, that your mentioned link to the Go code block is available in containerd 1.6.7: https://github.com/containerd/containerd/releases/tag/v1.6.7

Although we are on containerd 1.6.6 at AWS EKS.

[mkilchhofer]

Created two different Azure clusters to dig further into this:

containerd://1.6.4+azure-4

$ kubectl get nodes -o wide
NAME                                STATUS   ROLES   AGE   VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
aks-nodepool1-28766286-vmss000000   Ready    agent   26h   v1.23.12   10.115.34.8    <none>        Ubuntu 18.04.6 LTS   5.4.0-1098-azure   containerd://1.6.4+azure-4
(..)

$ kubectl exec -ti xyz-7c96875865-c4kqt -- sh
/app $ ps -ef
PID   USER     TIME  COMMAND
    1 baseuser  0:00 ./Post.xyz.Service
   36 baseuser  0:00 sh
   42 baseuser  0:00 ps -ef

/app $ cd /tmp/
/tmp $ /diag/dotnet-dump collect --process-id 1 --type full

Writing full to /tmp/core_20230125_163813
[createdump] Problem suspending threads: ptrace(ATTACH, 1) FAILED Operation not permitted (1)
[createdump] Failure took 0ms

/tmp $

containerd://1.6.15+azure-1

$ kubectl get nodes -o wide
NAME                                STATUS   ROLES   AGE    VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
aks-nodepool1-22805140-vmss000000   Ready    agent   116s   v1.23.12   10.115.36.9    <none>        Ubuntu 18.04.6 LTS   5.4.0-1100-azure   containerd://1.6.15+azure-1

$ kubectl exec -ti xyz-5c7bf5d5c9-bvn7b -- sh
/tmp $ /diag/dotnet-dump collect --process-id 1 --type full

Writing full to /tmp/core_20230125_165250
Complete
/tmp $

Both tests shared the exact same Kubernetes DeploymentSpec:

Expand DeploymentSpec

apiVersion: apps/v1
kind: Deployment
metadata:
labels:
  app: xyz
name: xyz
spec:
replicas: 1
selector:
  matchLabels:
    app: xyz
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 25%
template:
  metadata:
    labels:
      app: xyz
  spec:
    containers:
    - env:
      - name: ASPNETCORE_ENVIRONMENT
        value: Production
      - name: KUBERNETES_NAMESPACE
        valueFrom:
          fieldRef:
            fieldPath: metadata.namespace
      image: xxx.post.ch/xxx/requestbroadcaster:latest
      imagePullPolicy: Always
      name: requestbroadcaster
      ports:
      - containerPort: 5000
        protocol: TCP
      resources:
        requests:
          memory: "100Mi"
          cpu: "500m"
        limits:
          memory: "500Mi"
          cpu: "1000m"
      livenessProbe:
          httpGet:
              path: /health/live
              port: 5000
              scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
          timeoutSeconds: 3
      readinessProbe:
          httpGet:
              path: /health/ready
              port: 5000
              scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
          timeoutSeconds: 5
      securityContext:
        capabilities:
          # add:
          #   - SYS_PTRACE
          drop:
            - ALL
        seccompProfile:
          type: RuntimeDefault
      volumeMounts:
        - mountPath: /tmp
          name: tmp
    volumes:
      - emptyDir: {}
        name: tmp
    securityContext:
      runAsUser: 1000
      runAsGroup: 1000
      fsGroup: 1000

It's definitively a container runtime "issue" and is fixed in a newer version.

Conclusion: @msallin needs to wait for our Kubernetes platform update or add more privileges as a workaround until we upgraded to at least containerd 1.6.15 (or .7 as mentioned in the releases page above)

@msallin feel free to close this issue here. Thanks for all your support @noahfalk @hoyosjs @tommcdon