Can't use tmpfs as runroot for containers

[cwrau]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Cannot use tmpfs as runroot for containers

Steps to reproduce the issue:

1. Have $XDG_RUNTIME_DIR as tmpfs

   df -h $XDG_RUNTIME_DIR
   Filesystem      Size  Used Avail Use% Mounted on
   tmpfs           3.2G  300K  3.2G   1% /run/user/1000
   

2. Configure [storage].runroot:

   File: /home/cwr/.config/containers/storage.conf
   ───────────────────────────────────────────────
   [storage]
   runroot = "$XDG_RUNTIME_DIR/containers/storage"
   

Describe the results you received:

podman ps --all
Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at 
"/home/cwr/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

Describe the results you expected:
A working podman

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.6
Git Commit:   a0d478edea7f775b7ce32f8eb1a01e75374486cb
Built:        Tue Dec  8 22:48:23 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.22, commit: 9c34a8663b85e479e0c083801e89a2b2835228ed'
  cpus: 12
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: steve
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
  kernel: 5.9.13-arch1-1
  linkmode: dynamic
  memFree: 544743424
  memTotal: 33405747200
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc92
      commit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
      spec: 1.0.2-dev
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 0
  swapTotal: 0
  uptime: 119h 17m 7.6s (Approximately 4.96 days)
registries:
  search:
  - hub.4allportal.net
  - docker.io
store:
  configFile: /home/cwr/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.10.1
        fuse-overlayfs: version 1.3
        FUSE library version 3.10.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/cwr/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 52
  runRoot: /run/user/1000/containers
  volumePath: /home/cwr/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607464103
  BuiltTime: Tue Dec  8 22:48:23 2020
  GitCommit: a0d478edea7f775b7ce32f8eb1a01e75374486cb
  GoVersion: go1.15.6
  OsArch: linux/amd64
  Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

Name            : podman
Version         : 2.2.1-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/libpod
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : cni-plugins  conmon  containers-common  device-mapper  iptables  libseccomp  runc  slirp4netns  libsystemd  fuse-overlayfs  libgpgme.so=11-64
Optional Deps   : podman-docker: for Docker-compatible CLI
                  btrfs-progs: support btrfs backend devices [installed]
                  catatonit: --init flag support
                  crun: support for unified cgroupsv2
Required By     : podman-compose
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 79.09 MiB
Packager        : Morten Linderud <foxboron@archlinux.org>
Build Date      : Tue 08 Dec 2020 10:48:23 PM CET
Install Date    : Wed 09 Dec 2020 03:37:51 AM CET
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[vrothberg]

Can you set mountopt = "nodev,metacopy=on" in the your storage.conf? And, maybe, mount_program = "/usr/bin/fuse-overlayfs".

[rhatdan]

@giuseppe @rhvgoyal Does overlay work on tmpfs?

[cwrau]

I set

[storage]
runroot = "$XDG_RUNTIME_DIR/containers/storage"
[storage.options.overlay]
mountopt = "nodev,metacopy=on"
mount_program = "/usr/bin/fuse-overlayfs" // also tried without this

Still doesn't work, it returns the same error:

Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/home/cwr/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

[rhatdan]

You also need to set

driver="overlay"

[cwrau]

This works, but the storage folder doesn't exist, even after writing something in the container

podman-run docker.io/centos:7
[root@6ba502cdddb0 /]# echo a > a
[root@6ba502cdddb0 /]# 

drwx------   - cwr cwr 2020-12-18 14:41 /run/user/1000/containers/
drwx------   - cwr cwr 2020-12-18 14:41 ├── overlay/
drwx------   - cwr cwr 2020-12-18 14:41 ├── overlay-containers/
drwx------   - cwr cwr 2020-12-18 14:41 │  └── 6ba502cdddb0e606264d98dd8e830a3a1d723c20bc218124f3226bdaa032551e/
drwx------   - cwr cwr 2020-12-18 14:41 │     └── userdata/
.rw-r--r--   0 cwr cwr 2020-12-18 14:41 │        ├── .containerenv
.rw-r--r--   7 cwr cwr 2020-12-18 14:41 │        ├── conmon.pid
.rw-r--r--  12 cwr cwr 2020-12-18 14:41 │        ├── hostname
.rw-r--r--  97 cwr cwr 2020-12-18 14:41 │        ├── hosts
.rw-r--r--   0 cwr cwr 2020-12-18 14:41 │        ├── oci-log
.rw-r--r--   7 cwr cwr 2020-12-18 14:41 │        ├── pidfile
.rw-r--r-- 160 cwr cwr 2020-12-18 14:41 │        └── resolv.conf
drwx------   - cwr cwr 2020-12-18 14:41 ├── overlay-layers/
.rw------- 216 cwr cwr 2020-12-18 14:41 │  ├── mountpoints.json
.rw-r--r--  64 cwr cwr 2020-12-18 14:41 │  └── mountpoints.lock
drwx------   - cwr cwr 2020-12-18 14:41 └── overlay-locks/

Am I misunderstanding this config?

runroot=""                                                                                            
  container storage run dir (default: "/var/run/containers/storage")                                  
  Default directory to store all temporary writable content created by container storage programs.    
  The rootless runroot path supports environment variable substitutions (ie. $HOME/containers/storage)

Shouldn't this config cause the runtime to be there?

The mounts seem to still be in my home folder:

/bin/cat $XDG_RUNTIME_DIR/containers/overlay-layers/mountpoints.json
[{"id":"4fca09a09450b9fa1466da874d68398726bf5ec24298f0c1d707999da11e69ff","path":"/home/cwr/.local/share/containers/storage/overlay/4fca09a09450b9fa1466da874d68398726bf5ec24298f0c1d707999da11e69ff/merged","count":1}]

[giuseppe]

can you look under /home/cwr/.local/share/containers/storage ?

[cwrau]

That's where it is, but shouldn't it be in $XDG_RUNTIME_DIR/containers/storage?

[giuseppe]

no. The storage files are stored under graphroot. $XDG_RUNTIME_DIR/containers is only for volatile files.

Usually XDG_RUNTIME_DIR is on tmpfs, so it would be very expensive to have the storage there.

[cwrau]

As I understand, the storage files, like the images/layers itself should be in the graphroot.

But I thought runroot, temporary writable content, meant stuff like the overlayfs upper/work layer?

Am I wrong about that?

If so, how can I move that into my tmpfs?

[rhatdan]

Why do you want them on your tmpfs? The upper/work directories can grow quite large and run out of space on a tmpfs mount.
these directories are in the graphroot as well.

[cwrau]

For speed 😁

I don't need a lot of storage during runtime, I'd rather have speed.

I mostly manually mount, --tmpfs, memory onto known runtime folders for improved speed, so I just thought a config option would make my life easier 👍️

Also, I got 32GiB, often have more than 10GiB free 😅

So, there is no setting for runtime only? Only the complete graphroot?

So, what does runroot mean then?

[giuseppe]

runroot is used to store the state of running containers.

Another possibility if you'd like to keep the upper layers on tmpfs is to use an additional RO store. The RO store can be placed on the persistent storage, while the main store is on the tmpfs

[cwrau]

Mh, read-only doesn't sound like a drop-in configuration 😅

I think I'll stick with the current system, it's not like it's slow 😁

Thanks for your help though! 😊