Rootless podman fails to start up after switching storage driver

[debarshiray]

/kind bug

Description

If I remove all my podman and buildah containers and images, ~/.local/share/containers, and then change the storage driver in ~/.config/containers/storage.conf, subsequent attempts at using rootless podman error out:

$ podman info
could not get runtime: mkdir /var/run/containers: permission denied

$ podman --log-level=debug info
INFO[0000] running as rootless                          
DEBU[0000] Initializing boltdb state at /home/rishi/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /home/rishi/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Set libpod namespace to ""                   
ERRO[0000] could not get runtime: mkdir /var/run/containers: permission denied

My ~/.config/containers/libpod.conf:

volume_path = "/home/rishi/.local/share/containers/storage/volumes"
image_default_transport = "docker://"
runtime_path = ["/usr/bin/runc", "/usr/sbin/runc", "/usr/local/bin/runc", "/usr/local/sbin/runc", "/sbin/runc", "/bin/runc", "/usr/lib/cri-o-runc/sbin/runc"]
conmon_path = ["/usr/libexec/podman/conmon", "/usr/libexec/crio/conmon", "/usr/local/lib/podman/conmon", "/usr/local/libexec/crio/conmon", "/usr/bin/conmon", "/usr/sbin/conmon", "/usr/lib/crio/bin/conmon"]
conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"]
cgroup_manager = "cgroupfs"
init_path = "/usr/libexec/podman/catatonit"
static_dir = "/home/rishi/.local/share/containers/storage/libpod"
tmp_dir = "/run/user/1000/libpod/tmp"
max_log_size = -1
no_pivot_root = false
cni_config_dir = "/etc/cni/net.d/"
cni_plugin_dir = ["/usr/libexec/cni", "/usr/lib/cni", "/usr/local/lib/cni", "/opt/cni/bin"]
infra_image = "k8s.gcr.io/pause:3.1"
infra_command = "/pause"
enable_port_reservation = true
label = true
num_locks = 2048

My initial ~/.config/containers/storage.conf:

[storage]
  driver = "overlay"
  runroot = "/run/user/1000"
  graphroot = "/home/rishi/.local/share/containers/storage"
  [storage.options]
    mount_program = "/usr/bin/fuse-overlayfs"

Later I change it to:

RunRoot = "/run/user/1000"
GraphRoot = "/home/rishi/.local/share/containers/storage"
GraphDriverName = "vfs"
GraphDriverOptions = []

Additional information you deem important (e.g. issue happens only occasionally):

To recover:

$ rm -rf ~/.local/share/containers
$ rm ~/.config/containers/*.conf

Output of podman version:

Version:       0.12.2-dev
Go Version:    go1.11.2
Git Commit:    "64627d910b2113fa2de6b949d846af379e75e305"
Built:         Thu Jan 10 15:48:59 2019
OS/Arch:       linux/amd64

Output of podman info:

host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-0.12.1.2-1.git9551f6b.fc29.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: 67ab7549b44484cc3f201d7bb2b58b922f8edc24'
  Distribution:
    distribution: fedora
    version: "29"
  MemFree: 1735098368
  MemTotal: 8127053824
  OCIRuntime:
    package: runc-1.0.0-66.dev.gitbbb17ef.fc29.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6+dev
      commit: ead425507b6ba28278ef71ad06582df97f2d5b5f
      spec: 1.0.1-dev
  SwapFree: 4130848768
  SwapTotal: 4133482496
  arch: amd64
  cpus: 4
  hostname: kolache
  kernel: 4.19.12-301.fc29.x86_64
  os: linux
  rootless: true
  uptime: 3h 20m 25.21s (Approximately 0.12 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/rishi/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/rishi/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
  ImageStore:
    number: 0
  RunRoot: /run/user/1000

[mheon]

I suspect this is related to #2076

@giuseppe PTAL

[mheon]

It seems like runroot is being overridden at some point, and it's trying /var/run/containers instead of /run/user/1000 - when we debugged this yesterday, it seemed like the database was getting created with the bad /var/run/containers value

[mheon]

I can't reproduce on 0.12.1.2, which means this has to be from a recent PR.

[giuseppe]

I think this is caused by: #2076

now all the defaults are not set when the ~/.config/containers/storage.conf file exists

But the configuration file:

RunRoot = "/run/user/1000"
GraphRoot = "/home/rishi/.local/share/containers/storage"
GraphDriverName = "vfs"
GraphDriverOptions = []

Doesn't look correct. You need to use the correct format

[mheon]

@giuseppe Good catch!
@debarshiray TOML is case-sensitive, and I think all of these options are all lowercase, which might well be the problem

[giuseppe]

@debarshiray you probably want:

[storage]
driver = "vfs"
runroot = "/run/user/1000"
graphroot = "/home/rishi/.local/share/containers/storage"

[mheon]

@giuseppe @rhatdan
After the storage file has been loaded, if runroot or graphroot are still the empty string, should we restore defaults at that point to avoid situations like this?

[rhatdan]

Yes I think so.

[debarshiray]

But the configuration file:
...
Doesn't look correct. You need to use the correct format

podman seems to switch between the lower case and camel case formats all by itself. eg., if I rm ~/.config/containers/*.conf, then after the next invocation of rootless podman, I sometimes get one format, and at other times another.

[debarshiray]

Yes, #2131 fixes this for me. Thanks @giuseppe