additionalimagestores Not Working as Expected

[kasualkeef]

Issue Description

I have been able to successfully follow the guide for setting up a read-only image store as described here:
https://www.redhat.com/sysadmin/image-stores-podman

Unfortunately, the store does not seem read-only at all. If the user does not have write permissions to the files on the image store, he is unable to build images using a store image as the FROM base.

We are running rootless Podman on RHEL 8 with all of the latest patches from the RHEL repos.

Steps to reproduce the issue

Steps to reproduce the issue

1. Pull down images with the --root flag as described here.

   • This must be done as a rootless user or the in-container permissions will be completely broken

2. chmod the files so that other users can read the files

   • We have the force_mask='shared' storage.conf setting but this still doesn't seem to affect the file mode on the pulled files (they're still 0700)

3. At this stage, another rootless user on the system can point additionalimagestores to the image store directory and do simple things like view the images and run a container instance. However, image builds do not work correctly unless the rootless user has full write permissions to the image store.

    ## Pull Image into RO Store
    [rootless.user@rhel8dev01 ~]$ podman pull --root /data/image-cache/ harbor.domain.com/dod-ironbank/ironbank/redhat/ubi/ubi8
    Trying to pull harbor.domain.com/dod-ironbank/ironbank/redhat/ubi/ubi8:latest...
    Getting image source signatures
    Copying blob 1ea03711adea done  
    Copying blob e5dd65eaa632 done  
    Copying config b7c4db37ab done  
    Writing manifest to image destination
    b7c4db37ab5f29a75921f41120161a998149ac52f4605b3acd46325c08f903ba
    [rootless.user@rhel8dev01 ~]$ ls -l /data/image-cache/
    total 8
    drwx------. 2 rootless.user admins   27 Feb 12 07:24 libpod
    drwx------. 5 rootless.user admins  185 Feb 12 07:24 overlay
    drwx------. 2 rootless.user admins   29 Feb 12 07:24 overlay-containers
    drwx------. 3 rootless.user admins  116 Feb 12 07:24 overlay-images
    drwx------. 2 rootless.user admins 4096 Feb 12 07:24 overlay-layers
    -rw-r--r--. 1 rootless.user admins   64 Feb 12 07:24 storage.lock
    -rw-r--r--. 1 rootless.user admins    0 Feb 12 07:24 userns.lock
    [rootless.user@rhel8dev01 ~]$ chmod -R a+rX /data/image-cache/
    
    ## View RO Images
    [rootless.user@rhel8dev01 ~]$ su - other.rootless.user
    [other.rootless.user@rhel8dev01 test]$ podman images
    REPOSITORY                                                TAG         IMAGE ID      CREATED     SIZE        R/O
    harbor.example.com/dod-ironbank/ironbank/redhat/ubi/ubi8  latest      b7c4db37ab5f  2 days ago  246 MB      true
    
    ## Test Image Build
    [other.rootless.user@rhel8dev01 test]$ cat Dockerfile 
    FROM harbor.example.com/dod-ironbank/ironbank/redhat/ubi/ubi8
    
    RUN echo foo
    [other.rootless.user@rhel8dev01 test]$ podman build -f Dockerfile . -t test_build:latest
    STEP 1/2: FROM harbor.example.com/dod-ironbank/ironbank/redhat/ubi/ubi8
    STEP 2/2: RUN echo foo
    foo
    COMMIT test_build:latest
    Error: committing container for step {Env:[container=oci PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[echo foo] Flags:[] Attrs:map[] Message:RUN echo foo Original:RUN echo foo}: copying layers and metadata for container "3c689cd9949639fa75e3da707a574fe529c3004a56c9476d4fb2dc295f9ec353": initializing source containers-storage:ubi8-working-container: extracting layer "f8f29dbe2320223cd8c3e725967c24b24f1cce2bbb8c88ba42e715ecf76b0de6": chown /data/image-cache/overlay/ed9962cc48ff67290ea405d51a5cfa81069243fae21ff51857a754a17c2fbbaf/diff: operation not permitted
    
    ## Give User Write Perms and Try Build Again
    [root@rhel8dev01 ~]# chown -R other.rootless.user /data/image-cache/
    [root@rhel8dev01 ~]# su - other.rootless.user
    [other.rootless.user@rhel8dev01 ~]$ cd dev/test/
    [other.rootless.user@rhel8dev01 test]$ podman build -f Dockerfile . -t test_build:latest
    STEP 1/2: FROM harbor.example.com/dod-ironbank/ironbank/redhat/ubi/ubi8
    STEP 2/2: RUN echo foo
    foo
    COMMIT test_build:latest
    --> d71e8c5a6cee
    Successfully tagged localhost/test_build:latest
    d71e8c5a6ceea8858b99d3612f81ac2b603598d12f8c2c80cf43bd9423b8ccc2
    [other.rootless.user@rhel8dev01 ~]$ 
   

Describe the results you received

I see permissions errors on container image builds based on the RO image cache.

Describe the results you expected

I expect the additionalimagestores images to be read-only and useful for container image builds.

podman info output

[rootless.user@rhel8dev01 test]$ podman info
host:
  arch: amd64
  buildahVersion: 1.31.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.8-1.module+el8.9.0+21243+a586538b.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 64375b77dc0eda8f5a71b9208e041bc1e36eac81'
  cpuUtilization:
    idlePercent: 99.14
    systemPercent: 0.1
    userPercent: 0.76
  cpus: 12
  databaseBackend: boltdb
  distribution:
    distribution: '"rhel"'
    version: "8.9"
  eventLogger: file
  freeLocks: 2048
  hostname: rhel8dev01
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1941000000
      size: 1
    - container_id: 1
      host_id: 2918049
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1941000003
      size: 1
    - container_id: 1
      host_id: 2918049
      size: 65536
  kernel: 4.18.0-513.11.1.el8_9.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 6334828544
  memTotal: 10438213632
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns:
      package: podman-plugins-4.6.1-8.module+el8.9.0+21243+a586538b.x86_64
      path: /usr/libexec/cni/dnsname
      version: |-
        CNI dnsname plugin
        version: 1.3.1
        commit: unknown
    package: containernetworking-plugins-1.3.0-8.module+el8.9.0+21243+a586538b.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: runc
    package: runc-1.1.12-1.module+el8.9.0+21243+a586538b.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.12
      spec: 1.0.2-dev
      go: go1.20.12
      libseccomp: 2.5.2
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/user/1941000003/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_SYS_CHROOT,CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.module+el8.9.0+21243+a586538b.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 6462369792
  swapTotal: 6462369792
  uptime: 4h 11m 6.00s (Approximately 0.17 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/users/rootless.user/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /data/image-cache
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.12-1.module+el8.9.0+21243+a586538b.x86_64
      Version: |-
        fusermount3 version: 3.3.0
        fuse-overlayfs: version 1.12
        FUSE library version 3.3.0
        using FUSE kernel interface version 7.26
  graphRoot: /data/podman_users/rootless.user/share/containers/storage
  graphRootAllocated: 268287614976
  graphRootUsed: 2173411328
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /data/scratch/
  imageStore:
    number: 2
  runRoot: /data/podman_users/rootless.user/containers
  transientStore: false
  volumePath: /data/podman_users/rootless.user/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1707224641
  BuiltTime: Tue Feb  6 08:04:01 2024
  GitCommit: ""
  GoVersion: go1.20.12
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

The storage.conf configuration is as follows:

[storage]
  driver = "overlay"
  runroot = "/data/podman_users/$USER/containers"
  graphroot = "/data/podman_users/$USER/share/containers/storage"
  rootless_storage_path = "/data/podman_users/$USER/share/containers/storage"
  [storage.options]
    additionalimagestores = ['/data/image-cache']   
    size = ""
    remap-uids = ""
    remap-gids = ""
    ignore_chown_errors = ""
    remap-user = ""
    remap-group = ""
    skip_mount_home = ""
    mount_program = "/usr/bin/fuse-overlayfs"
    mountopt = ""
    [storage.options.overlay]
      ignore_chown_errors = ""
      mountopt = ""
      mount_program = "/usr/bin/fuse-overlayfs"
      force_mask = "shared"
      size = ""
      skip_mount_home = ""

Additional information

No response

[rhatdan]

@giuseppe PTAL

[giuseppe]

opened a PR: #1828

[kasualkeef]

Good morning,
I just built the latest podman RC (5.0.0-rc5) which should have the storage library containing the fix (1.53.0). However, I'm still seeing the same chown error when building an image based on one in the additional image stores cache:

Do I also need to build buildah with the new storage library?

[rootless.user@dev01 test]$ podman build -t test -f Dockerfile
STEP 1/2: FROM harbor.example.com/dod-ironbank/ironbank/redhat/ubi/ubi8
STEP 2/2: RUN echo 'hello world'
hello world
COMMIT test
Error: committing container for step {Env:[container=oci PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[echo 'hello world'] Flags:[] Attrs:map[] Message:RUN echo 'hello world' Heredocs:[] Original:RUN echo 'hello world'}: copying layers and metadata for container "ab406d6793451713acf8476dd7170dce401f2b54c7794da6e260e7f4dfc010d0": initializing source containers-storage:ubi8-working-container: extracting layer "f94c9202039b49b4cfb32bb4238b397aa3c1e2d9c263e7b0088721c78f4f00c5": chown /shared-scratch/image-cache/overlay/ed2dfe10b654b557c3cb9dce9aa2f5095a5a8d4c1ac49752cd9d6e63c879aec2/merged: operation not permitted

Podman Info:

[rootless.user@dev01 test]$ podman info
host:
  arch: amd64
  buildahVersion: 1.35.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.8-1.module+el8.9.0+21243+a586538b.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 64375b77dc0eda8f5a71b9208e041bc1e36eac81'
  cpuUtilization:
    idlePercent: 97.15
    systemPercent: 0.46
    userPercent: 2.38
  cpus: 12
  databaseBackend: sqlite
  distribution:
    distribution: rhel
    version: "8.9"
  eventLogger: file
  freeLocks: 2048
  hostname: dev01
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1941000000
      size: 1
    - container_id: 1
      host_id: 493217
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1941000003
      size: 1
    - container_id: 1
      host_id: 493217
      size: 65536
  kernel: 4.18.0-513.18.1.el8_9.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 252215296
  memTotal: 10438201344
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.module+el8.9.0+21243+a586538b.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-2.module+el8.9.0+21243+a586538b.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: runc
    package: runc-1.1.12-1.module+el8.9.0+21243+a586538b.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.12
      spec: 1.0.2-dev
      go: go1.20.12
      libseccomp: 2.5.2
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-g137ce01-1.x86_64
    version: |
      pasta 2024_02_20.1e6f92b-51-g137ce01
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_SYS_CHROOT,CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.module+el8.9.0+21243+a586538b.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 3634360320
  swapTotal: 6462369792
  uptime: 5h 19m 39.00s (Approximately 0.21 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/users/rootless.user/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.ignore_chown_errors: "true"
    overlay.imagestore: /shared-scratch/image-cache
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.12-1.module+el8.9.0+21243+a586538b.x86_64
      Version: |-
        fusermount3 version: 3.3.0
        fuse-overlayfs: version 1.12
        FUSE library version 3.3.0
        using FUSE kernel interface version 7.26
  graphRoot: /data/podman_users/rootless.user/share/containers/storage
  graphRootAllocated: 268287614976
  graphRootUsed: 2183630848
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /data/scratch/
  imageStore:
    number: 7
  runRoot: /data/podman_users/rootless.user/containers
  transientStore: false
  volumePath: /data/podman_users/rootless.user/share/containers/storage/volumes
version:
  APIVersion: 5.0.0-rc5
  Built: 1709904410
  BuiltTime: Fri Mar  8 08:26:50 2024
  GitCommit: ""
  GoVersion: go1.20.12
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.0-rc5

[giuseppe]

does the /shared-scratch/image-cache/overlay/ed2dfe10b654b557c3cb9dce9aa2f5095a5a8d4c1ac49752cd9d6e63c879aec2/merged directory exist?

[giuseppe]

part of the fix is to never remove that directory. But if your storage was created without the fix in place, that directory might not be present.

[kasualkeef]

The directory did exist but was seeded with an older Podman version. I rebuilt the image cache using Podman 5.0.0-rc5 and still get the chown error when building an image based one in the cache.

[rootless.user2@dev01 test]$ podman build -t test -f Dockerfile .
STEP 1/2: FROM harbor.example.com/dod-ironbank/ironbank/redhat/ubi/ubi8
STEP 2/2: RUN echo 'hello world'
hello world
COMMIT test
Error: committing container for step {Env:[container=oci PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[echo 'hello world'] Flags:[] Attrs:map[] Message:RUN echo 'hello world' Heredocs:[] Original:RUN echo 'hello world'}: copying layers and metadata for container "de6057ed6a1fcad927c82f30fe77c414196e6ce4e68ea2322515ed74b4fad28f": initializing source containers-storage:ubi8-working-container: extracting layer "252d17bdddbec50adae92b95f703eec608d494f7613cbac0d0d3a435c347a48f": chown /shared-scratch/image-cache/overlay/ed2dfe10b654b557c3cb9dce9aa2f5095a5a8d4c1ac49752cd9d6e63c879aec2/merged: operation not permitted
[rootless.user2@dev01 test]$ ls -l /shared-scratch/image-cache/overlay/ed2dfe10b654b557c3cb9dce9aa2f5095a5a8d4c1ac49752cd9d6e63c879aec2/
total 8
drwxr-xr-x. 7 rootless.user1 rootless.user1 65 Mar 11 08:21 diff
-rwxr-xr-x. 1 rootless.user1 rootless.user1 26 Mar 11 08:21 link
-rwxr-xr-x. 1 rootless.user1 rootless.user1 28 Mar 11 08:21 lower
drwxr-xr-x. 2 rootless.user1 rootless.user1  6 Mar 11 08:21 merged
drwxr-xr-x. 2 rootless.user1 rootless.user1  6 Mar 11 08:21 work

[giuseppe]

then it looks like we might still get the EPERM, as we attempt the chown.

Could you please try if #1858 solve the problem?

[kasualkeef]

Good news! I built 5.0.0-rc5 with the storage library from github.com/giuseppe/storage@create-merged-only-if-it-doesnot-exist (v1.19.2-0.20240311145129-0fbd068b4eee) and I no longer get the chown error building an image based on the cache.

I still need to test a bit more but it looks good.