Cannot get container stats on rootless container

[marusak]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Calling varlink GetContainerStats on rootless container returns Link not found error.

Steps to reproduce the issue:

1. $ varlink call unix:/run/user/1000/podman/io.podman/io.podman.ListContainers
   {
     "containers": [
   {
     "command": [
       "/bin/bash"
     ],
     "containerrunning": true,
     "createdat": "2019-10-15T10:55:01+02:00",
     "id": "ec038ff11a1431c76c70da48a0b0452b9ec6f851a7b9b8a0db934c5662d5551f",
     "image": "docker.io/library/fedora:30",
     "imageid": "e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea0160759c637999",
     "labels": {
       "maintainer": "Clement Verna <cverna@fedoraproject.org>"
     },
     "mounts": [
       {
         "destination": "/proc",
         "options": [
           "nosuid",
           "noexec",
           "nodev"
         ],
         "source": "proc",
         "type": "proc"
       },
       {
         "destination": "/dev",
         "options": [
           "nosuid",
           "strictatime",
           "mode=755",
           "size=65536k"
         ],
         "source": "tmpfs",
         "type": "tmpfs"
       },
       {
         "destination": "/dev/pts",
         "options": [
           "nosuid",
           "noexec",
           "newinstance",
           "ptmxmode=0666",
           "mode=0620",
           "gid=5"
         ],
         "source": "devpts",
         "type": "devpts"
       },
       {
         "destination": "/dev/mqueue",
         "options": [
           "nosuid",
           "noexec",
           "nodev"
         ],
         "source": "mqueue",
         "type": "mqueue"
       },
       {
         "destination": "/sys",
         "options": [
           "nosuid",
           "noexec",
           "nodev",
           "ro"
         ],
         "source": "sysfs",
         "type": "sysfs"
       },
       {
         "destination": "/sys/fs/cgroup",
         "options": [
           "rprivate",
           "nosuid",
           "noexec",
           "nodev",
           "relatime",
           "ro"
         ],
         "source": "cgroup",
         "type": "cgroup"
       }
     ],
     "names": "upbeat_ellis",
     "namespaces": {
       "cgroup": "4026531835",
       "ipc": "4026533304",
       "mnt": "4026533302",
       "net": "4026532653",
       "pid": "29571",
       "pidns": "4026533305",
       "user": "4026532650",
       "uts": "4026533303"
     },
     "rootfssize": 246026159,
     "runningfor": "13m27.748950048s",
     "rwsize": 0,
     "status": "running"
      }
     ]
   }
   

2. $ varlink call unix:/run/user/1000/podman/io.podman/io.podman.GetContainerStats '{"name": "ec038ff11a1431c76c70da48a0b0452b9ec6f851a7b9b8a0db934c5662d5551f"}'`
   Call failed with error: io.podman.ErrorOccurred
   {
     "reason": "Link not found"
   }
   

Describe the results you received:
Link not found error

Describe the results you expected:
Something like: (this is doing the same but with root container)

    {
  "container": {
    "block_input": 20852736,
    "block_output": 24576,
    "cpu": 5.062848779281613169954e-09,
    "cpu_nano": 19886000,
    "id": "5a3c596ad2bd7f2a123f9178ab27346c13c08fa80fdd183b38e4d41797a9c343",
    "mem_limit": 16426668032,
    "mem_perc": 5.448311235465040647252e-02,
    "mem_usage": 8949760,
    "name": "upbeat_payne",
    "net_input": 698,
    "net_output": 10907,
    "pids": 1,
    "system_nano": 188802000
  }
}

Additional information you deem important (e.g. issue happens only occasionally):
This haven't worked on Fedora 30 due to CGroupsV1, but I migrated to CGroupsV2 (I am on Fedora 31).

Output of podman version:

Version:            1.6.1
RemoteAPI Version:  1
Go Version:         go1.13.1
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.1
  podman version: 1.6.1
host:
  BuildahVersion: 1.11.2
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.1-1.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.1, commit: 5e0eadedda9508810235ab878174dca1183f4013'
  Distribution:
    distribution: fedora
    version: "31"
  MemFree: 8753004544
  MemTotal: 16426668032
  OCIRuntime:
    package: crun-0.10.2-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.10.2
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: localhost
  kernel: 5.3.4-300.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 46m 20.42s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/mmarusak/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.6.4-2.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.6.4
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /home/mmarusak/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 15
  RunRoot: /run/user/1000
  VolumePath: /home/mmarusak/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.6.1-5.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

[mheon]

Hm. 1.6.1 on FC31 means Cgroups v2. That's probably our issue.

We fixed psgo for podman top but stats does not call into that. It uses our internal cgroups library, which theoretically should work with v2.

[mheon]

@marusak Can you confirm that this doesn't happen with 1.6.1 on FC30 (really, anything that's not FC31)?

[marusak]

@marusak Can you confirm that this doesn't happen with 1.6.1 on FC30 (really, anything that's not FC31)?

$ cat /etc/os-release | grep PRETTY
PRETTY_NAME="Fedora 30 (Thirty)"

$ rpm -q podman
podman-1.6.1-2.fc30.x86_64

$ varlink call unix:/run/user/1000/podman/io.podman/io.podman.GetContainerStats '{"name": "a622eefed5e3c37271d9226000c41cd56381e418d5bcef815b011ec851d096f5"}'
Call failed with error: io.podman.ErrRequiresCgroupsV2ForRootless
{
  "reason": "rootless containers cannot report container stats"
}

A bit different version (1.6.1-2 vs 1.6.1-5) hope it is not issue. If it is I can update and retry.

[mheon]

OK, that sounds like an error we'd expect. We expect it to not work on any system without V2. The issue seems to be that it isn't working even with V2 enabled.

[baude]

patching coming