User can delete images, even when used by root containers (with additionalimagestores) #13003

klaus-scheitterer-mw · 2022-01-25T11:51:26Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
When adding the graphRoot of a user podman to the additionalimagestores of root podman images from user podman are deleted even if there are existing containers in root podman.
Maybe a known issue, but the shortcomings are not very specific in this regard.

Steps to reproduce the issue:

add graphRoot of user podman (e.g. /home/vagrant/.local/share/containers/storage) to additionalimagestores in /etc/containers/storage.conf
pull image in user podman: podman pull hello-world
create container as root with image pulled by rootless podman: sudo podman create --name test hello-world
delete image as user: podman rmi hello-world

Describe the results you received:
image is deleted:

$ podman rmi hello-world
Untagged: docker.io/library/hello-world:latest
Deleted: feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412

root container is still present but (understandably) unable to start:

$ sudo podman start test --attach
WARN[0000] Can't stat lower layer "/var/lib/containers/storage/overlay/l/7HS76F2P5N73FDUKUQAOJA3WI5" because it does not exist. Going through storage to recreate the missing symlinks.
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/baba52f31d3482f8b577007a60ad901c142cdaff82f189a26f80f3459c1a63e7/merged: invalid argument
Error: unable to start container c526e7015ade6e638f1b122ec6554e9e9a8ddaafed83af8a792347065fdc1342: error mounting storage for container c526e7015ade6e638f1b122ec6554e9e9a8ddaafed83af8a792347065fdc1342: error creating overlay mount to /var/lib/containers/storage/overlay/baba52f31d3482f8b577007a60ad901c142cdaff82f189a26f80f3459c1a63e7/merged, mount_data="nodev,lowerdir=/var/lib/containers/storage/overlay/l/7HS76F2P5N73FDUKUQAOJA3WI5,upperdir=/var/lib/containers/storage/overlay/baba52f31d3482f8b577007a60ad901c142cdaff82f189a26f80f3459c1a63e7/diff,workdir=/var/lib/containers/storage/overlay/baba52f31d3482f8b577007a60ad901c142cdaff82f189a26f80f3459c1a63e7/work,context=\"system_u:object_r:container_file_t:s0:c26,c661\"": no such file or directory

Describe the results you expected:
Image deletion should fail, as it would if the user podman had containers created with that image, e.g.

Error: Image used by <container-id>: image is in use by a container

Output of podman version:

Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.15.14
Built:        Mon Aug 30 15:46:46 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.27-2.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 2
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: fedora33.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.14.18-100.fc33.x86_64
  linkmode: dynamic
  memFree: 931053568
  memTotal: 2054926336
  ociRuntime:
    name: crun
    package: crun-1.0-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.0
      commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc33.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 3174031360
  swapTotal: 3174031360
  uptime: 38m 6.12s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/vagrant/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/vagrant/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/vagrant/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630356406
  BuiltTime: Mon Aug 30 15:46:46 2021
  GitCommit: ""
  GoVersion: go1.15.14
  OsArch: linux/amd64
  Version: 3.3.1

Output of sudo podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.27-2.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 2
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: fedora33.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.14.18-100.fc33.x86_64
  linkmode: dynamic
  memFree: 928993280
  memTotal: 2054926336
  ociRuntime:
    name: crun
    package: crun-1.0-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.0
      commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc33.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 3174031360
  swapTotal: 3174031360
  uptime: 39m 25.4s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /home/vagrant/.local/share/containers/storage
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630356406
  BuiltTime: Mon Aug 30 15:46:46 2021
  GitCommit: ""
  GoVersion: go1.15.14
  OsArch: linux/amd64
  Version: 3.3.1

Package info (e.g. output of rpm -q podman or apt list podman):

$ dnf info podman
Last metadata expiration check: 0:02:49 ago on Tue 25 Jan 2022 05:43:52 AM CST.
Installed Packages
Name         : podman
Epoch        : 3
Version      : 3.3.1
Release      : 1.fc33
Architecture : x86_64
Size         : 49 M
Source       : podman-3.3.1-1.fc33.src.rpm
Repository   : @System
From repo    : updates
Summary      : Manage Pods, Containers and Container Images
URL          : https://podman.io/
License      : ASL 2.0

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):
Tested in Vagrant box:

Vagrant.configure("2") do |config|
  config.vm.box = "generic/fedora33"

  config.vm.provision "install podman", type: "shell", inline: <<-SHELL
    dnf install -y podman
  SHELL

end

The text was updated successfully, but these errors were encountered:

rhatdan · 2022-01-25T14:30:07Z

The issue is that the "additionalStore" knows nothing about its images being used in other stores, and there is no way to know. So I think this should just be documented in troubleshooting.

klaus-scheitterer-mw · 2022-01-27T07:37:36Z

Thanks for the feedback.
I also looked at the storage code and agree. Simply no way in general to know about what other podman instances are doing with your image store.
I try to find the time make an PR to describe this constraint in troubleshooting.md.

NINJAFURRY · 2022-02-10T16:00:45Z

I am also facing the same issue but in the case of pod creation.

github-actions · 2022-03-13T00:06:31Z

A friendly reminder that this issue had no activity for 30 days.

Closes: containers#13003 [NO NEW TESTS NEEDED] Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe · 2022-03-22T12:15:27Z

PR here: #13590

Closes: containers#13003 [NO NEW TESTS NEEDED] Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Jan 25, 2022

github-actions bot added the stale-issue label Mar 13, 2022

giuseppe added a commit to giuseppe/libpod that referenced this issue Mar 22, 2022

[CI:DOCS] troubleshooting: document rm in image stores

07999b2

Closes: containers#13003 [NO NEW TESTS NEEDED] Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe mentioned this issue Mar 22, 2022

[CI:DOCS] troubleshooting: document rm in image stores #13590

Merged

openshift-merge-robot closed this as completed in #13590 Mar 22, 2022

keonchennl pushed a commit to gcalin/podman that referenced this issue Mar 24, 2022

[CI:DOCS] troubleshooting: document rm in image stores

5b40e44

Closes: containers#13003 [NO NEW TESTS NEEDED] Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

keonchennl pushed a commit to gcalin/podman that referenced this issue Mar 29, 2022

[CI:DOCS] troubleshooting: document rm in image stores

153ea6b

Closes: containers#13003 [NO NEW TESTS NEEDED] Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

gbraad pushed a commit to gbraad-redhat/podman that referenced this issue Jul 13, 2022

[CI:DOCS] troubleshooting: document rm in image stores

0ead13c

Closes: containers#13003 [NO NEW TESTS NEEDED] Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

User can delete images, even when used by root containers (with additionalimagestores) #13003

User can delete images, even when used by root containers (with additionalimagestores) #13003

klaus-scheitterer-mw commented Jan 25, 2022

rhatdan commented Jan 25, 2022

klaus-scheitterer-mw commented Jan 27, 2022

NINJAFURRY commented Feb 10, 2022

github-actions bot commented Mar 13, 2022

giuseppe commented Mar 22, 2022

User can delete images, even when used by root containers (with additionalimagestores) #13003

User can delete images, even when used by root containers (with additionalimagestores) #13003

Comments

klaus-scheitterer-mw commented Jan 25, 2022

rhatdan commented Jan 25, 2022

klaus-scheitterer-mw commented Jan 27, 2022

NINJAFURRY commented Feb 10, 2022

github-actions bot commented Mar 13, 2022

giuseppe commented Mar 22, 2022