Listing container changes tries to write/mount additionalimagestores as upperdir

[leahneukirchen]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

additionalimagestores is supposed to provide a read-only storage of unpacked layers.
However, using it breaks the /libpod/containers/{name}/changes endpoint, as it tries to
write or do writeable mounts inside the additionalimagestores.

Steps to reproduce the issue:

1. Pull alpine:latest into additionalimagestores

2. Create a container that uses layers there:

# podman run -d alpine:latest sleep 100000
a1b2160977183b03033f8680d271219b6a97437c46da2de3fdfbd7b2f122f436

# podman exec a1b2160977183b03033f8680d271219b6a97437c46da2de3fdfbd7b2f122f436 mount |grep overlay
overlay on / type overlay (rw,relatime,lowerdir=/nfs/podman/overlay/l/VZE7XWEKYHPEUQZH4PSF66E6WQ,upperdir=/var/lib/containers/storage/overlay/f3c1b9c35107f0b5850ecae69ff65229a4a96d8bcaf1fded25ac1dd740ee942d/diff,workdir=/var/lib/containers/storage/overlay/f3c1b9c35107f0b5850ecae69ff65229a4a96d8bcaf1fded25ac1dd740ee942d/work,metacopy=on)

Until this step, the additionalimagestores is properly used as lowerdir only.

3. Query /libpod/containers/{name}/changes and trigger an error:

# curl --unix-socket /var/run/podman/podman.sock http://localhost/v3.4.1/libpod/containers/a1b2160977183b03033f8680d271219b6a97437c46da2de3fdfbd7b2f122f436/changes
{"cause":"error creating overlay mount to /nfs/podman/overlay/e2eb06d8af8218cfec8210147357a68b7e13f7c485b991c288c2d01dc228bb68/merged, mount_data=\"nodev,metacopy=on,lowerdir=/nfs/podman/overlay/e2eb06d8af8218cfec8210147357a68b7e13f7c485b991c288c2d01dc228bb68/empty,upperdir=/nfs/podman/overlay/e2eb06d8af8218cfec8210147357a68b7e13f7c485b991c288c2d01dc228bb68/diff,workdir=/nfs/podman/overlay/e2eb06d8af8218cfec8210147357a68b7e13f7c485b991c288c2d01dc228bb68/work\": invalid argument","message":"error creating overlay mount to /nfs/podman/overlay/e2eb06d8af8218cfec8210147357a68b7e13f7c485b991c288c2d01dc228bb68/merged, mount_data=\"nodev,metacopy=on,lowerdir=/nfs/podman/overlay/e2eb06d8af8218cfec8210147357a68b7e13f7c485b991c288c2d01dc228bb68/empty,upperdir=/nfs/podman/overlay/e2eb06d8af8218cfec8210147357a68b7e13f7c485b991c288c2d01dc228bb68/diff,workdir=/nfs/podman/overlay/e2eb06d8af8218cfec8210147357a68b7e13f7c485b991c288c2d01dc228bb68/work\": invalid argument","response":500}

In dmesg we see:

overlayfs: filesystem on '/nfs/podman/overlay/e2eb06d8af8218cfec8210147357a68b7e13f7c485b991c288c2d01dc228bb68/diff' not supported as upperdir

Describe the results you received:

The additionalimagestores was used as an upperdir, creating an error.

Describe the results you expected:

I expected to see the changes I did to the container (in this case, none).

If an upperdir is needed for this operation (not sure why), it should be created in graphRoot, just like for the actual container.

Output of podman version:

Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.6
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 2
  distribution:
    codename: focal
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: runner-jmxdd-staging
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.0-1043-gcp
  linkmode: dynamic
  logDriver: journald
  memFree: 3919024128
  memTotal: 7817076736
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 0
  swapTotal: 0
  uptime: 21m 11.36s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 4
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /nfs/podman
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 108
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Not tested.

Additional environment details (AWS, VirtualBox, physical, etc.):

/nfs/podman is a NFS3 read-only mount.

[leahneukirchen]

The problem is overlay.Driver get using overlay.Driver dir, which returns paths within the additional image store, and then using that dir to create work, diff, merged dirs inside.

[rhatdan]

Could you write up a step by step environment on how to set this up?

[leahneukirchen]

Any read-only image store will do. You need to use overlayfs or fuse-overlay as a backend:

docker run --privileged --rm -ti -v imagestore:/imagestore mgoltzsche/podman /bin/sh
# podman version
Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.10
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

# podman --root /imagestore pull docker.io/alpine:latest
# mkdir /imagestore-ro
# mount -o bind /imagestore /imagestore-ro
# mount -o remount,ro,bind /imagestore /imagestore-ro
# touch /imagestore-ro/foo
touch: /imagestore-ro/foo: Read-only file system
# vi /etc/containers/storage.conf
... add additionalimagestores=["/imagestore-ro"] ...

# apk add curl
# podman system service -t 0 &
# curl --unix /var/run/podman/podman.sock http://localhost/_ping
OK
# podman --remote run -d docker.io/alpine:latest sleep 100000
6623d4cb0ef36104102b75a355710b61180292daed31f65a2514e30dcd2b290b
# curl --unix /var/run/podman/podman.sock http://localhost/v3.4.1/libpod/containers/6623d4cb0ef36104102b75a355710b61180292daed31f65a2514e30dcd2b290b/changes
ERRO[0114] error unmounting /imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/merged: invalid argument 
{"cause":"error creating overlay mount to /imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/merged, mount_data=\"nodev,fsync=0,lowerdir=/imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/empty,upperdir=/imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/diff,workdir=/imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/work\": using mount program /usr/local/bin/fuse-overlayfs: fuse-overlayfs: cannot open workdir: No such file or directory\n: exit status 1","message":"error creating overlay mount to /imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/merged, mount_data=\"nodev,fsync=0,lowerdir=/imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/empty,upperdir=/imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/diff,workdir=/imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/work\": using mount program /usr/local/bin/fuse-overlayfs: fuse-overlayfs: cannot open workdir: No such file or directory\n: exit status 1","response":500}

Alternative way, without remote:

# podman run -d docker.io/alpine:latest sleep 100000
e160f0e546d288bd9deb569600d1ccd51bc82880513ae47ab7ceb6528c721dbf
# podman diff e160f0e546d288bd9deb569600d1ccd51bc82880513ae47ab7ceb6528c721dbf
ERRO[0000] error unmounting /imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/merged: invalid argument 
Error: error creating overlay mount to /imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/merged, mount_data="nodev,fsync=0,lowerdir=/imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/empty,upperdir=/imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/diff,workdir=/imagestore-ro/overlay/8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759/work": using mount program /usr/local/bin/fuse-overlayfs: fuse-overlayfs: cannot open workdir: No such file or directory
: exit status 1

[leahneukirchen]

Likewise, podman image diff docker.io/library/alpine will fail.

[rhatdan]

Thanks I hope to get some time to look into this. @giuseppe do you have some time?

[leahneukirchen]

I tried fixing it in linked PR, perhaps it's a start.

[giuseppe]

is the patch for containers/storage enough to address this issue?

[leahneukirchen]

This patch is enough, yes.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[leahneukirchen]

Was fixed in containers/storage#1123