rootless podman not using conmon_path from /etc/containers/libpod.conf

[sysrich]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

running podman run hello-world as a non-root user produces the following error message

error creating libpod runtime: could not find a working conmon binary (configured options: [/usr/libexec/podman/conmon /usr/libexec/crio/conmon /usr/local/lib/podman/conmon /usr/local/libexec/crio/conmon /usr/bin/conmon /usr/sbin/conmon /usr/lib/crio/bin/conmon]): invalid argument

The container obviously does not start

/etc/containers/libpod.conf has conmon_path set to /usr/lib/podman/bin/conmon, which is the correct location for conmon on all *SUSE distributions.

The above error message clearly implies that podman has not read this parameter and instead is only looking in the default locations.

My expectation was that rootless podman should inherit its parameters from /etc/containers/libpod.conf and only be overriden by a ~.config/containers/libpod.conf if it is manually created by the user.

Steps to reproduce the issue:

1. Install podman on openSUSE Tumbleweed version 20190115 or later. Alternatively install Kubic with the microos system role (the default) - podman is the default container runtime there.

2. run podman run hello-world as a non-root user

3. There is no step 3

Output of podman version:

Version:       1.0.0
Go Version:    go1.11.4
OS/Arch:       linux/amd64

Output of podman info:

host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-1.0.0-1.1.x86_64
    path: /usr/lib/podman/bin/conmon
    version: 'conmon version 1.12.0-dev, commit: '
  Distribution:
    distribution: '"opensuse-tumbleweed-kubic"'
    version: "20190115"
  MemFree: 11926519808
  MemTotal: 16487182336
  OCIRuntime:
    package: runc-1.0.0~rc6-2.1.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 4
  hostname: r2d2
  kernel: 4.20.0-1-default
  os: linux
  rootless: false
  uptime: 7h 35m 10.36s (Approximately 0.29 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: btrfs
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Build Version: 'Btrfs v4.19.1 '
    Library Version: "102"
  ImageStore:
    number: 3
  RunRoot: /var/run/containers/storage

[giuseppe]

yes, at the moment we don't honor /etc/containers/libpod.conf for rootless containers, but use directly the version in the user config directory.

Is conmon installed at some path that the default libpod configuration doesn't know about?

[sysrich]

/usr/lib/podman/bin/conmon is in the default libpod configuration documented here: https://github.com/containers/libpod/blob/master/libpod.conf

But the error message implies that podman 1.0.0 is not honoring that default libpod configuration either and looking at a different list of paths somehow.

[vrothberg]

/usr/lib/podman/bin/conmon is in the default libpod configuration documented here: https://github.com/containers/libpod/blob/master/libpod.conf

But the error message implies that podman 1.0.0 is not honoring that default libpod configuration either and looking at a different list of paths somehow.

I agree. It seems that when generating ~/.config/containers/libpod.conf the hard-coded defaults are used instead of copying specific values from /etc/containers/libpod.conf.

[vrothberg]

Podman should definitely do that for config options. openSUSE is shipping a libpod.conf tailored specifically for openSUSE-specific paths to avoid costly lookups for binaries.

[sysrich]

I agree. It seems that when generating ~/.config/containers/libpod.conf the hard-coded defaults are used instead of copying specific values from /etc/containers/libpod.conf.

I agree, and it seems to me the hard coded defaults are different from the defaults written in the default libpod.conf from https://github.com/containers/libpod/blob/master/libpod.conf

[giuseppe]

Podman should definitely do that for config options.

I am not sure about it. We should be careful and cherry-pick only the options that can be inherited safely from the global configuration. Some of them would override the defaults tailored for the rootless mode.

[giuseppe]

maybe we should maintain a white list for those that must be inherited from /etc/containers/libpod.conf the first time the rootless configuration is created

[vrothberg]

ACK. We shouldn't blindly copy but need to pick the ones that make sense.

[afbjorklund]

I had the same issue, and had to patch libpod because of it (lack of rootless configuration)

sed -e 's|"/etc/cni|"/usr/local/etc/cni|' -i libpod.conf
sed -e 's|"/etc/cni|"/usr/local/etc/cni|' -i libpod/runtime.go
sed -e 's|/usr/libexec/|/usr/local/lib/|' -i libpod.conf
sed -e 's|/usr/libexec/|/usr/local/lib/|' -i libpod/runtime.go
sed -e 's|/usr/share/|/usr/local/share/|' -i libpod/runtime.go

Otherwise it is not able to find CNI, which means that rootless podman is not able to start up.

Unfortunately it also means it is "dirty", so configuration would be preferred.

Git Commit: "4e0c0ecbc383531cd1b38db9027583974a72070d-dirty"

[rhatdan]

We should definitely get the executable paths from libpod.conf

[giuseppe]

opened a PR here: #2222

@sysrich would it be possible for you to try if it solves the issue you had?

[afbjorklund]

Would it be possible to add no_pivot_root and cni_config_dir too ?

I think that cgroup_manager is determined automatically (?), so it is OK...

# CGroup Manager - valid values are "systemd" and "cgroupfs"
cgroup_manager = "cgroupfs"

# Whether to use chroot instead of pivot_root in the runtime
no_pivot_root = true

# Directory containing CNI plugin configuration files
cni_config_dir = "/usr/local/etc/cni/net.d/"

[mheon]

The systemd cgroup manager doesn't work at all with rootless, so we shouldn't bother sourcing that - it needs to stay unconditionally set to "cgroupfs"

The other two would be nice to have, though

[giuseppe]

I can add cni_plugin_dir and no_pivot_root, although rootless containers do not support CNI plugins, so it won't make any difference for now.

cni_config_dir doesn't probably make sense. Even if there will be rootless CNI plugins, the configuration for rootless will be different than from the host

[afbjorklund]

The cni_plugin_dir was OK, since it looked in multiple paths (including /usr/local/lib/cni)

I seem to recall some error from CNI, but maybe that was just normal podman and not rootless ?

[afbjorklund]

Got some other error now: ERRO[0001] slirp4netns failed
Guess I will look into that particular podman error separately.

The other error was (as I suspected) when running as root:
ERRO[0000] error configuring network namespace for container 7dcf3329d7bd6f6372967fcfffaf78bac904246910005bcafc4c4d0260397bca: Missing CNI default network

So no_pivot_root would be enough.