Running "podman system reset" deleted my entire $HOME directory

[gg7]

Issue Description

podman info was reporting that the vfs storage driver was in use, even though my kernel is new enough (6.1) and I have /usr/bin/fuse-overlayfs present.

From what I read that's because I already had vfs layers, so I ran podman system reset as my own (non-root) user, which worked mostly okay (I couldn't get the default to be overlay so I had to create ~/.config/containers/storage.conf, but that's for another ticket I guess).

Then I decided to also run podman system reset as root because I remember running a few containers as root years ago. As a result, podman proceeded to delete the entire /root/ folder.

Steps to reproduce the issue

I have been able to reproduce the issue. For example, let's create /root/.ssh/ and run podman system reset:

gentoo ~ # ls -alh
total 16K
drwxr-xr-x  3 root root 4.0K 2023-04-20 12:39:49 ./
drwxr-xr-x 21 root root 4.0K 2023-04-20 12:35:32 ../
drwxr-xr-x  2 root root 4.0K 2023-04-20 12:39:49 .ssh/
-rw-------  1 root root 1.2K 2023-04-20 12:39:49 .viminfo
hostname ~ # podman system reset                                                                           
WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all networks
        - all build cache
        - all machines
        - all volumes
Are you sure you want to continue? [y/N] y
gentoo ~ # ls -alh
total 0
gentoo ~ # pwd
/root
gentoo ~ # ls -al /root                                                                                  
total 48K
drwxr-xr-x 10 root root 4.0K 2023-04-20 12:43:10 ./
drwxr-xr-x 21 root root 4.0K 2023-04-20 12:40:12 ../
drwxr-xr-x  3 root root 4.0K 2023-04-20 12:40:12 .config/
drwx------  2 root root 4.0K 2023-04-20 12:43:10 mounts/
drwx------  3 root root 4.0K 2023-04-20 12:43:10 overlay/
drwx------  2 root root 4.0K 2023-04-20 12:43:10 overlay-containers/
drwx------  2 root root 4.0K 2023-04-20 12:43:10 overlay-images/
drwx------  2 root root 4.0K 2023-04-20 12:43:10 overlay-layers/
drwx------  2 root root 4.0K 2023-04-20 12:43:10 overlay-locks/
drwx------  2 root root 4.0K 2023-04-20 12:43:10 tmp/
-rw-r--r--  1 root root    3 2023-04-20 12:43:10 defaultNetworkBackend
-rw-r--r--  1 root root   64 2023-04-20 12:43:10 storage.lock
-rw-r--r--  1 root root    0 2023-04-20 12:43:10 userns.lock
gentoo ~ #

Maybe /root/ is intact, but podman mounted something on top of it? Unfortunately that's not the case:

gentoo ~ # mount | grep root
gentoo ~ # 

Describe the results you received

podman deleted /root/.

Describe the results you expected

I expected podman to delete all containers/pods/images/networks/build cache/machines/volumes, but not my /root/ folder.

podman info output

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: app-containers/conmon-2.0.30
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.30, commit: v2.0.30'
  cpuUtilization:
    idlePercent: 98.95
    systemPercent: 0.66
    userPercent: 0.38
  cpus: 6
  distribution:
    distribution: gentoo
    version: "2.13"
  eventLogger: journald
  hostname: gentoo
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.1.22-gentoo-dist
  linkmode: dynamic
  logDriver: journald
  memFree: 45350371328
  memTotal: 67280412672
  networkBackend: cni
  ociRuntime:
    name: crun
    package: app-containers/crun-1.4.5
    path: /usr/bin/crun
    version: |-
      crun version 1.4.5
      commit: c381048530aa750495cf502ddb7181f2ded5b400
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: app-containers/slirp4netns-1.2.0
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.4
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 4h 9m 25.00s (Approximately 0.17 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
    PullFromMirror: ""
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /root
  graphRootAllocated: 498425044992
  graphRootUsed: 203847122944
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /root
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1679773711
  BuiltTime: Sat Mar 25 19:48:31 2023
  GitCommit: 814b7b003cc630bf6ab188274706c383f9fb9915
  GoVersion: go1.20.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

Bare metal.

Additional information

AFAIK I am using the defaults as far as storage is concerned:

gentoo ~ # ls -alh /etc/containers/
total 44K
drwxr-xr-x   3 root root 4.0K 2023-04-20 12:34:10 ./
drwxr-xr-x 115 root root  12K 2023-04-20 14:33:22 ../
drwxr-xr-x   2 root root 4.0K 2023-03-25 20:00:01 registries.d/
-rw-r--r--   1 root root 4.6K 2023-04-20 11:17:15 policy.json
-rw-r--r--   1 root root 4.7K 2023-03-25 19:49:34 policy.json.example
-rw-r--r--   1 root root  959 2023-04-20 11:17:58 registries.conf
-rw-r--r--   1 root root  883 2023-03-25 19:49:34 registries.conf.example

In my opinion it's highly surprising for podman to delete /root using out-of-the-box configuration.

[Luap99]

graphRoot: /root

Podman was configured somehow to use /root. A podman system reset will delete the graph root so this eplains why /root is deleted.

Do you have a /usr/share/containers/storage.conf file?

[gg7]

Do you have a /usr/share/containers/storage.conf file?

I do not:

gentoo ~ # stat /usr/share/containers/storage.conf
stat: cannot statx '/usr/share/containers/storage.conf': No such file or directory
gentoo ~ #

The only storage.conf I have is for my own user (the one I created earlier today):

gentoo ~ # find / -name storage.conf
/home/gg7/.config/containers/storage.conf
gentoo ~ # 

[Luap99]

Do you have something under /var/lib/containers/, if you do please delete it and try to run podman info again it should default to this directly as storage location normally.

[gg7]

Do you have something under /var/lib/containers/?

Yes, I do:

gentoo ~ # find /var/lib/containers/
/var/lib/containers/
/var/lib/containers/cache
/var/lib/containers/.keep_app-containers_podman-0
/var/lib/containers/storage
/var/lib/containers/storage/tmp
/var/lib/containers/storage/libpod
/var/lib/containers/storage/libpod/bolt_state.db

if you do please delete it and try to run podman info again it should default to this directly as storage location normally.

That works, thank you:

gentoo ~ # rm -r /var/lib/containers/cache /var/lib/containers/storage
gentoo ~ # find /var/lib/containers/
/var/lib/containers/
/var/lib/containers/.keep_app-containers_podman-0
gentoo ~ # podman info | grep graphRoot:
  graphRoot: /var/lib/containers/storage
gentoo ~ #

Looks like podman info has recreated /var/lib/containers/storage:

gentoo ~ # find /var/lib/containers/
/var/lib/containers/
/var/lib/containers/.keep_app-containers_podman-0
/var/lib/containers/storage
/var/lib/containers/storage/mounts
/var/lib/containers/storage/tmp
/var/lib/containers/storage/overlay
/var/lib/containers/storage/overlay/l
/var/lib/containers/storage/overlay/.has-mount-program
/var/lib/containers/storage/defaultNetworkBackend
/var/lib/containers/storage/libpod
/var/lib/containers/storage/libpod/bolt_state.db
/var/lib/containers/storage/storage.lock
/var/lib/containers/storage/userns.lock
/var/lib/containers/storage/overlay-layers
/var/lib/containers/storage/overlay-layers/layers.lock
/var/lib/containers/storage/overlay-images
/var/lib/containers/storage/overlay-images/images.lock
/var/lib/containers/storage/overlay-containers
/var/lib/containers/storage/overlay-containers/containers.lock

Even though I have fixed this particular computer, in my opinion this is still a bug which should be fixed. I have not created anything under /var/lib/containers/ by hand -- it must have been an earlier podman run. podman deleted /root through no fault of my own and this can happen to others too.

I would break down the issue as:

• bug 1 (root cause): podman picks a suboptimal default graphRoot (at the very least polluting root's home directory) just because /var/lib/containers/ is partially initialized (because /var/lib/containers/cache exists?)
• bug 2: podman system reset did not delete /var/lib/containers/cache and /var/lib/containers/storage for me. That command is also meant to delete $HOME/.config/containers/storage.conf , right? If it deletes config files, why not clean up the default location as well?
• bug 3: podman system reset deleted the whole /root folder, instead of /root/{storage,cache,...} (say, if someone had manually set graphRoot to /root). Why isn't the deletion code more defensive? If someone sets graphRoot: /, this will nuke their whole system. At the very least the warning message can print which directory will be deleted.

Just my two cents, I am not a podman expert. And thanks again for the help!

[rhatdan]

We have no idea how this happened or why it would delete /root. If you created a reproducer we would certainly fix it. We have never heard of something like this happening. and have no idea how it happened if the root directory in storage.conf did not point at that directory.