How to configure podman to use volumes quota on XFS storage

[fdb-git]

I want to set a quota to podman volumes (using an XFS storage mounted with prjquota option) so I configured as documented here:

root@odroid:~# cat /etc/projects 
100000:/storage/containers/graphroot/overlay
200000:/storage/containers/graphroot/volumes

root@odroid:~# cat /etc/projid 
storage:100000
volumes:200000

When I create the first volume it's all ok:

root@odroid:~# podman volume create --opt o=size=100M volume1
volume1

root@odroid:~# xfs_quota -x -c "report -a -h" /storage/containers
Project quota on /storage/containers (/dev/mapper/luks-73764eef-ecc9-4323-aadc-9940738aff36)
                        Blocks              
Project ID   Used   Soft   Hard Warn/Grace   
---------- --------------------------------- 
#0           128K      0      0  00 [------]
storage        4K      0      0  00 [------]
volumes         0      0      0  00 [------]
#200001         0  95.4M  95.4M  00 [------]

But when I create the second volume it does not create a second project id but only updates the quota of the first:

root@odroid:~# podman volume create --opt o=size=200M volume2
volume2
root@odroid:~# xfs_quota -x -c "report -a -h" /storage/containers
Project quota on /storage/containers (/dev/mapper/luks-73764eef-ecc9-4323-aadc-9940738aff36)
                        Blocks              
Project ID   Used   Soft   Hard Warn/Grace   
---------- --------------------------------- 
#0           128K      0      0  00 [------]
storage        4K      0      0  00 [------]
volumes         0      0      0  00 [------]
#200001         0 190.7M 190.7M  00 [------]

What am I doing wrong?

The following is my configuration using Podman provided by Ubuntu 22.04.
I had already tried with podman 4.6.2. using the Kubic packages but I got the same behavior.

root@odroid:~# podman info
host:
  arch: arm64
  buildahVersion: 1.23.1
  cgroupControllers:
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 4
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: odroid
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.9.337-13
  linkmode: dynamic
  logDriver: journald
  memFree: 2585001984
  memTotal: 3889238016
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 0
  swapTotal: 0
  uptime: 2h 4m 36.78s (Approximately 0.08 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /storage/containers/graphroot
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /storage/containers/runroot
  volumePath: /storage/containers/graphroot/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  OsArch: linux/arm64
  Version: 3.4.4

[eriksjolund]

I tried out the same but in the opposite order (where the second volume is smaller than the first volume).
The same thing happened. The quota was set to the second volume.

disclaimer: I haven't used quota with XFS before. I might be doing something wrong.

[jake-desco]

I am actually wondering how this is intended to be used as well but for the quotas created for individual containers. Containers are often ephemeral and usually referred to by name; the project IDs assigned to them are not predictable so I'm not sure how one is meant to assign a quota for one without monitoring for every start and dynamically setting the quotas. One might actually need to set the quota before the container starts. Perhaps container run should take a project ID option to allow users to control this.

Then I think that calculating a project ID based on the base project that /var/lib/containers/storage/overlay (or presumably whatever the graph root is set to) is mapped to, is potentially problematic. While maybe useful when the administrator has reserved a range of project IDs for Podman, it causes issues when quotas were enabled for other purposes on a host. Take the example of :

1:/
2:/data
3:/swap
4:/var/lib/systemd/coredump
5:/var/log
6:/var/tmp

.. which does not have a specific project set up for Podman. When I create containers, Podman uses Project=1 for its base project ID and then creates containers that use Project 2 and Project 3 etc -- using quota that belongs to other uses within the system. This prevented podman from starting on some hosts recently as swap is generally full.

I would suggest that Podman's manipulation and use of XFS quotas be an opt-in by default on machines with quotas enabled. I expect the magic instantiation of projects and especially the calculation of IDs is not what most system administrators expect..

[yvf]

Current behavior around assignment of project IDs basically means that podman MUST be given its own XFS filesystem. It is simply unsafe to share the filesystem with any other usage, as the ID-assigning algo assumes it owns the ID space, and does not check for existing usage when assigning a new ID.

[fdb-git]

The use case is to create nonvolatile volumes without using additional external storage infrastructure.

In particular, I refer to the instructions found here:
https://docs.podman.io/en/latest/markdown/podman-volume-create.1.html#quotas

I expected each volume to have its own xfs project-id and associated quota, but it does not.