vfs is used for rootless containers

[benbro]

Issue Description

I've installed the official podman package 4.3.1 on Debain 12 and created a rootless container.
The vfs storage driver is used instead of native overlayfs. This blog post suggests that native overlayfs should be used by default on supported podman version and Linux kernel.

The rootless_tutorial is confusing or outdated because it recommends to use fuse-overlayfs instead of native overlayfs.

Do I need to add ~/.config/containers/storage.conf and manually set the storage driver or should it get selected automatically?
Do I need to use this:

[storage]
driver = "overlay"

Or this?

[storage]
driver = "overlay"
[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

Steps to reproduce the issue

Steps to reproduce the issue

1. Install podman on debain 12 with "sudo apt-get install podman"
2. create and run a container
3. check the storage driver with podman info -f {{.Store.GraphDriverName}}

Describe the results you received

vfs

Describe the results you expected

overlay

podman info output

host:
  arch: amd64
  buildahVersion: 1.28.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 96.95
    systemPercent: 0.94
    userPercent: 2.11
  cpus: 2
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: journald
  hostname: dev
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.0-9-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 1067044864
  memTotal: 6213353472
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+b1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 1022357504
  swapTotal: 1022357504
  uptime: 1h 28m 14.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/me/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/me/.local/share/containers/storage
  graphRootAllocated: 56795291648
  graphRootUsed: 16524029952
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 13
  runRoot: /run/user/1000/containers
  volumePath: /home/me/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 02:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

No

### Additional environment details

_No response_

### Additional information

_No response_

[giuseppe]

it should get selected automatically, unless you already have containers on a vfs store.

Could you run podman system reset -f (be careful this will wipe out your entire containers storage) and attach the output of podman --log-level debug pull alpine?

[benbro]

Output of podman --log-level debug pull alpine after running podman system reset -f

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman --log-level debug pull alpine) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/me/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver vfs                       
DEBU[0000] Using graph root /home/me/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/me/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/me/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman --log-level debug pull alpine) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/me/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver vfs                       
DEBU[0000] Using graph root /home/me/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/me/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/me/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "vfs"   
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
DEBU[0000] Successfully loaded 1 networks               
DEBU[0000] Podman detected system restart - performing state refresh 
INFO[0000] Setting parallel job count to 7              
DEBU[0000] Pulling image alpine (policy: always)        
DEBU[0000] Looking up image "alpine" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/shortnames.conf" 
DEBU[0000] Trying "docker.io/library/alpine:latest" ... 
DEBU[0000] Trying "localhost/alpine:latest" ...         
DEBU[0000] Trying "docker.io/library/alpine:latest" ... 
DEBU[0000] Trying "alpine" ...                          
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate docker.io/library/alpine:latest for alpine 
DEBU[0000] parsed reference into "[vfs@/home/me/.local/share/containers/storage+/run/user/1000/containers]docker.io/library/alpine:latest" 
DEBU[0000] Resolved "alpine" as an alias (/etc/containers/registries.conf.d/shortnames.conf) 
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
DEBU[0000] Copying source image //alpine:latest to destination image [vfs@/home/me/.local/share/containers/storage+/run/user/1000/containers]docker.io/library/alpine:latest 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "docker.io/library/alpine:latest" 
DEBU[0000] No credentials matching docker.io/library/alpine found in /run/user/1000/containers/auth.json 
DEBU[0000] No credentials matching docker.io/library/alpine found in /home/me/.config/containers/auth.json 
DEBU[0000] No credentials matching docker.io/library/alpine found in /home/me/.docker/config.json 
DEBU[0000] No credentials matching docker.io/library/alpine found in /home/me/.dockercfg 
DEBU[0000] No credentials for docker.io/library/alpine found 
DEBU[0000]  No signature storage configuration found for docker.io/library/alpine:latest, using built-in default file:///home/me/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io 
DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/manifests/latest 
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" 
DEBU[0001] Using blob info cache at /home/me/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0001] Source is a manifest list; copying (only) instance sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70 for current system 
DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/manifests/sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70 
DEBU[0002] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" 
DEBU[0002] IsRunningImageAllowed for image docker:docker.io/library/alpine:latest 
DEBU[0002]  Using default policy section                
DEBU[0002]  Requirement 0: allowed                      
DEBU[0002] Overall: allowed                             
DEBU[0002] Downloading /v2/library/alpine/blobs/sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67 
DEBU[0002] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67 
Getting image source signatures
DEBU[0003] Reading /home/me/.local/share/containers/sigstore/library/alpine@sha256=25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70/signature-1 
DEBU[0003] Not looking for sigstore attachments: disabled by configuration 
DEBU[0003] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0003] ... will first try using the original manifest unmodified 
DEBU[0003] Checking if we can reuse blob sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0003] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0003] Downloading /v2/library/alpine/blobs/sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3 
DEBU[0003] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3 
Copying blob 31e352740f53 [--------------------------------------] 0.0b / 3.2MiB
DEBU[0003] Detected compression format gzip             
Copying blob 31e352740f53 done  
Copying blob 31e352740f53 done  
DEBU[0004] Untar time: 0.131802986s                     
DEBU[0004] No compression detected                      
DEBU[0004] Compression change for blob sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67 ("application/vnd.docker.container.image.v1+json") not supported 
DEBU[0004] Using original blob without modification     
Copying config c1aabb73d2 done  
Writing manifest to image destination
Storing signatures
DEBU[0004] setting image creation date to 2023-06-14 20:41:59.079795125 +0000 UTC 
DEBU[0004] created new image ID "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" 
DEBU[0004] saved image metadata "{\"signatures-sizes\":{\"sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70\":[]}}" 
DEBU[0004] added name "docker.io/library/alpine:latest" to image "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" 
DEBU[0004] Pulled candidate docker.io/library/alpine:latest successfully 
DEBU[0004] Looking up image "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" in local containers storage 
DEBU[0004] Trying "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" ... 
DEBU[0004] parsed reference into "[vfs@/home/me/.local/share/containers/storage+/run/user/1000/containers]@c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" 
DEBU[0004] Found image "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" as "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" in local containers storage 
DEBU[0004] Found image "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" as "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" in local containers storage ([vfs@/home/me/.local/share/containers/storage+/run/user/1000/containers]@c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67) 
DEBU[0004] exporting opaque data as blob "sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" 
c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67
DEBU[0004] Called pull.PersistentPostRunE(podman --log-level debug pull alpine)

[Luap99]

@giuseppe Note this is an old podman (4.3.1). I think your storage changes went in after that. So the answer is use a newer podman or just configure it the config file like you shown, fuse-overlay is not needed since kernel 5.13.

If you change the config file you must podman system reset in order for it to get picked up.

[giuseppe]

good eye! Podman 4.3.1 is hitting this issue: containers/storage#1570

[benbro]

Please update the rootless_tutorial to indicate that fuse-overlay is not needed.

I don't think there is a reasonable way to update podman on debian/ubuntu. If I run into a bug after few minutes playing with podman I'll probably run into more bugs so I'll might as well just stick with docker until the next debian release 2 years from now.

[Luap99]

I wouldn't call that a bug. vfs works, it is just not the best driver is many use cases.

As for updating the docs, I agree PRs are always welcome.

[giuseppe]

there is a simple workaround: just force the overlay driver in the configuration file. Then of course, if you prefer to run containers as root, that is your choice.

[rhatdan]

Add a storage.conf file with

[storage]

# Default Storage Driver, Must be set for proper operation.
driver = "overlay"

Then do podman system reset

[benbro]

After adding driver="overlay" to the config podman does use it. Thank you.

My problems with this:

• My request to fix the rootless_tutorial was ignored. There is nothing more frustrating than following outdated docs.
• I don't have an upgrade path for podman on debian/ubuntu. This storage issue has an easy workaround but next issues might not and we'll be too invested to switch.
• podman is changing fast. Following a moving target with outdated tutorials and docs without an upgrade path just won't work.

[rhatdan]

Did you open a PR for the rootless_tutorial? Remember this is an upstream project, we attempt to work well with distributions, but some of them do not keep up. There is a HUGE amount of work going on and a ton of issues, RFEs and enhancments going on all the time. So if a PR or issue is not noticed, it is probably not intentional, but an issue does not guarantee someone is going to work on it, since there are so many other pressing needs.

The core team had a goal of only having 200 outstanding issues, but since Podman popularity has skyrocketed we can not keep up. See issues standing at 456 as we speak. Community help is required and always welcome.