--containall in Singularity will result in operation not permitted inside a Kubernetes pod

[stxue1]

When Singularity is called with --containall, since it has to separate the PID namespaces, it tries to create its own proc filesystem as it can't take the existing proc from the external container. When running in a Kubernetes pod the kernel doesn't seem to allow this though, and returns an operation not permitted error:

FATAL:   container creation failed: mount proc->/proc error: while mounting proc: can't mount proc filesystem to /proc: operation not permitted

I'm able to replicate this by launching a Kubernetes instance on AWS, launching a pod, and running singularity --verbose exec --pwd /mnt/miniwdl_task_container/work --containall --no-mount hostfs docker://ubuntu:latest ls inside the pod. The yaml I'm using for kubernetes is:

apiVersion: v1
kind: Pod
metadata:
  name: test-kubernetes-pod
spec:
  containers:
    - name: test-kubernetes-container
      image: quay.io/stxue/toil:5.13.0a1-d384d7d250cc8687525edcc04236f0ca19e01093-dirty-py3.10
      imagePullPolicy: Always
      env:
      args: ["sleep", "infinity"]
      resources:
        requests:
          cpu: 1000m
          memory: "1Gi"
          ephemeral-storage: "10Gi"
        limits:
          cpu: 1000m
          memory: "1Gi"
          ephemeral-storage: "10Gi"
      volumeMounts:
        - mountPath: /scratch
          name: scratch
  restartPolicy: Never
  volumes:
    - name: scratch
      emptyDir: {}

The default MiniWDL configuration has --containall as part of the singularity command, which causes issues in Toil when running it on Kubernetes. Maybe the default config options for singularity shouldn't have --containall or MiniWDL should detect if --containall works.

It looks like this issue is also related to this: apptainer/singularity#5857