Using `$UID` in `storage.conf` no longer works

[siepkes]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

(I've also tested and reproduced this on a clean Fedora 33 install in a VM)

After I did an Fedora 33 dnf update any rootless podman command will result in a single line error ERRO[0000] no such file or directory:

$ podman info
ERRO[0000] no such file or directory    

Running with verbose logging doesn't tell much more:

podman --log-level=trace info
INFO[0000] podman filtering at log level trace          
DEBU[0000] Called info.PersistentPreRunE(podman --log-level=trace info) 
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.36.0 Annotations:[] CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:cgroupfs ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/user/1000/libpod/tmp/events/events.log EventsLogger:journald HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: InfraCommand: InfraImage:k8s.gcr.io/pause:3.5 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/mnt/storage1/podman/containers/1000/libpod StopTimeout:10 TmpDir:/run/user/1000/libpod/tmp VolumePath:/mnt/storage1/podman/containers/1000/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/siepkes/.config/cni/net.d}} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /mnt/storage1/podman/containers/1000/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /mnt/storage1/podman/containers/1000 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /mnt/storage1/podman/containers/1000/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /mnt/storage1/podman/containers/1000/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend journald          
TRAC[0000] found runtime ""                             
TRAC[0000] found runtime ""                             
TRAC[0000] found runtime ""                             
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
ERRO[0000] no such file or directory                

The only non default change is in /etc/containers/storage.conf:

rootless_storage_path = "/mnt/storage1/podman/containers/$UID"

Wiping out the rootless_storage_path target and recreating it does not make any difference.

I should note that I also merged the config updates in storage.conf.rpmnew with the existing storage.conf.

Interestingly enough setting the rootless_storage_path back to it's original rootless_storage_path = "$HOME/.local/share/containers/storage" causes all podman commands to hang forever. strace doesn't reveal much what it hangs on:

newfstatat(AT_FDCWD, "/usr/lib64/qt-3.3/bin/fuse-overlayfs", 0xc0004aa1d8, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/share/Modules/bin/fuse-overlayfs", 0xc0004aa2a8, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/siepkes/.cargo/bin/fuse-overlayfs", 0xc0004aa378, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/local/bin/fuse-overlayfs", 0xc0004aa448, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/local/sbin/fuse-overlayfs", 0xc0004aa518, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/bin/fuse-overlayfs", {st_mode=S_IFREG|0755, st_size=97024, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/siepkes/.config/containers/storage.conf", 0xc0004aa6b8, 0) = -1 ENOENT (No such file or directory)
futex(0xc000098148, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT ILL TRAP ABRT BUS FPE SEGV TERM STKFLT CHLD PROF SYS RTMIN RT_1], NULL, 8) = 0
futex(0xc000577148, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT ILL TRAP ABRT BUS FPE SEGV TERM STKFLT CHLD PROF SYS RTMIN RT_1], NULL, 8) = 0
futex(0xc000098148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x55a5b4503bc8, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT ILL TRAP ABRT BUS FPE SEGV TERM STKFLT CHLD PROF SYS RTMIN RT_1], NULL, 8) = 0
futex(0xc00058e148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x55a5b4503bc8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x55a5b4503bc8, FUTEX_WAIT_PRIVATE, 0, NULL

Steps to reproduce the issue:

1. Install Fedora 33 (in a VM).
2. Disable SElinux
3. mkdir -p /test-storage/1000 && chown -R 1000:1000 /test-storage/1000 && chmod 700 /test-storage/1000
4. Configure rootless_storage_path to a /test-storage/$UID
5. Reboot
6. Update with dnf -y update
7. Reboot
8. Run podman info as user.

Describe the results you received:

ERRO[0000] no such file or directory

Describe the results you expected:

Aside from not getting an error I would at least expect the error to include the actual path on what it tries to open.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

ERRO[0000] no such file or directory                    

Output of podman info --debug:

ERRO[0000] no such file or directory

Output of sudo podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.20.1
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.27-2.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: t-800
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.11.16-200.fc33.x86_64
  linkmode: dynamic
  memFree: 27734810624
  memTotal: 33667719168
  ociRuntime:
    name: crun
    package: crun-0.19.1-2.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.19.1
      commit: 1535fedf0b83fb898d449f9680000f729ba719f5
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 21189615616
  swapTotal: 21189615616
  uptime: 11m 30.12s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 19
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.1.2
  Built: 1619097693
  BuiltTime: Thu Apr 22 15:21:33 2021
  GitCommit: ""
  GoVersion: go1.15.8
  OsArch: linux/amd64
  Version: 3.1.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.1.2-1.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical Fedora 33 install:

$ name -a
Linux t-800 5.11.16-200.fc33.x86_64 #1 SMP Wed Apr 21 16:08:37 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

SeLinux is disabled:

$ selinuxenabled 
$ echo $?
1

[siepkes]

I figured it out. Apparently the $UID variable substitution support in storage.conf broke in an podman update. Using /test-storage/1000 works but /test-storage/$UID does not work any more after a podman update (don't know which one).

[gdoepp]

Same for me with ubuntu 20.04 using driver overlay, after upgrade from 3.0 to 3.1.0. $USER or $HOME is not expanded either. I found not substituted variables in strace.
graphroot = os.ExpandEnv(graphroot)
rundir = os.ExpandEnv(rundir)
in vendor/github.com/containers/storage/drivers/overlay/overlay.go SupportsNativeOverlay() seems to help.

[mheon]

Can you upgrade to 3.1.2? This is fixed upstream.

…

On Sat, May 1, 2021 at 09:55 gdoepp ***@***.***> wrote: Same for me with ubuntu 20.04 using driver overlay, after upgrade from 3.0 to 3.1.0. $USER or $HOME is not expanded either. I found not substituted variables in strace. graphroot = os.ExpandEnv(graphroot) rundir = os.ExpandEnv(rundir) in vendor/github.com/containers/storage/drivers/overlay/overlay.go SupportsNativeOverlay() seems to help. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10181 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCAT4JW4B6AQQFDQLK3TLQB4HANCNFSM433726CQ> .

[siepkes]

@mheon Are you sure 3.1.2 would help for @gdoepp on Ubuntu? I also encountered this issue on 3.1.2 on Fedora 34:

$ rpm -q podman
podman-3.1.2-1.fc34.x86_64

[gdoepp]

$UID, $USER or $HOME are still not replaced in 3.1.2

This is what strace -f says:

[pid 668359] geteuid()                  = 0
[pid 668359] newfstatat(AT_FDCWD, "/mnt/storage/containers/storage-$UID/overlay/.has-mount-program", 0xc0004e3558, 0) = -1 ENOENT (Datei oder Verzeichnis nicht gefunden)
[pid 668359] newfstatat(AT_FDCWD, "/mnt/storage/containers/storage-$UID/overlay", 0xc0004e3628, 0) = -1 ENOENT (Datei oder Verzeichnis nicht gefunden)
[pid 668359] newfstatat(AT_FDCWD, "/mnt/storage/containers/storage-$UID/overlay", 0xc0004e36f8, 0) = -1 ENOENT (Datei oder Verzeichnis nicht gefunden)
[pid 668360] <... nanosleep resumed>NULL) = 0
[pid 668359] newfstatat(AT_FDCWD, "/mnt/storage/containers/storage-$UID",  <unfinished ...>

podman version (from Ubuntu 20.04):

Version:      3.1.2
API Version:  3.1.2
Go Version:   go1.15.2
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

[gdoepp]

Commit a19bc5f makes podman work for me, too.