“docker container top” fails with error regarding runc termination: "getting all container pids from cgroups caused: lstat : no such file or directory"

[readyready15728]

I've posted the same on the Docker Forums and not gotten an answer after a few days and, in addition to a thorough hunt on Google for similar reports, I also found nothing, so I'm pretty sure this is a novel report here. Forgive me if I overlooked something obvious to an expert, because I just got started with Docker.

Anyway, having been advised to do similar in the past, I'm going to replicate the full text here:

I have Docker set up, including rootless mode, on a CentOS 8 Digital Ocean droplet. I am very new to this subject matter and am following along with Docker in a Month of Lunches and am now at the point where the reader is instructed to type in:

docker container run --interactive --tty diamol/base

Then the reader is instructed to open up another terminal, which I did, and type in:

docker container top [container id]

The command spits out the following:

Error response from daemon: runc did not terminate successfully: container_linux.go:187: getting all container pids from cgroups caused: lstat : no such file or directory
: unknown

Because I am a Docker neophyte, my ability to assist anyone reading to diagnose the problem is limited beyond providing what is typically asked for; however, I think it’s worth noting that the same thing happened when I tried the alpine:latest image, meaning it’s not specific to the book image. It also seems worth pointing out that the same thing does not happen when I repeat the same procedure above but with sudo.

Now, for the rest, as stated, I am using CentOS 8 on Digital Ocean. I used the get.docker.com script and followed the instructions for rootless mode (with the minor and most likely unimportant deviation that I tailored the instructions to fish rather than bash; $DOCKER_HOST has still been exported properly).

Specifically, here is the content of /etc/redhat-release:

CentOS Linux release 8.4.2105

Here’s the output of uname -a:

Linux metalicus 4.18.0-305.7.1.el8_4.x86_64 #1 SMP Tue Jun 29 21:55:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

And last but not least here’s the output of docker version:

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:24 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:48 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.8
  GitCommit:        7eba5930496d9bbe375fdf71603e610ad737d2b2
 runc:
  Version:          1.0.0
  GitCommit:        v1.0.0-0-g84113ee
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

...and docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 19
  Running: 0
  Paused: 0
  Stopped: 19
 Images: 3
 Server Version: 20.10.7
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2
 runc version: v1.0.0-0-g84113ee
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
 Kernel Version: 4.18.0-305.7.1.el8_4.x86_64
 Operating System: CentOS Linux 8
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.775GiB
 Name: metalicus
 ID: LM7D:OFP7:HCZ3:BHLN:S2VG:GVCH:CSRH:WRSY:7AFM:GRQT:IPOA:CPBU
 Docker Root Dir: /home/readyready15728/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

I’d be perfectly happy to attempt to supply whatever other details are needed to resolve the problem, including taking action on the Droplet, which is still running. My suspicion is that WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode will point me in the right direction but I am submitting an issue anyway because the error message remains extremely opaque, and should be altered if my suspicions are correct.

[thaJeztah]

@AkihiroSuda ptal; is this related to the warning about cgroups in rootless-mode?

WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.

[AkihiroSuda]

Yes, docker top requires cgroup

Please see https://rootlesscontaine.rs/getting-started/common/cgroup2/ to enable rootless cgroup2