Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rootless podman not using conmon_path from /etc/containers/libpod.conf #2174

Closed
sysrich opened this issue Jan 17, 2019 · 16 comments

Comments

Projects
None yet
7 participants
@sysrich
Copy link
Contributor

commented Jan 17, 2019

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

running podman run hello-world as a non-root user produces the following error message

error creating libpod runtime: could not find a working conmon binary (configured options: [/usr/libexec/podman/conmon /usr/libexec/crio/conmon /usr/local/lib/podman/conmon /usr/local/libexec/crio/conmon /usr/bin/conmon /usr/sbin/conmon /usr/lib/crio/bin/conmon]): invalid argument

The container obviously does not start

/etc/containers/libpod.conf has conmon_path set to /usr/lib/podman/bin/conmon, which is the correct location for conmon on all *SUSE distributions.

The above error message clearly implies that podman has not read this parameter and instead is only looking in the default locations.

My expectation was that rootless podman should inherit its parameters from /etc/containers/libpod.conf and only be overriden by a ~.config/containers/libpod.conf if it is manually created by the user.

Steps to reproduce the issue:

  1. Install podman on openSUSE Tumbleweed version 20190115 or later. Alternatively install Kubic with the microos system role (the default) - podman is the default container runtime there.

  2. run podman run hello-world as a non-root user

  3. There is no step 3

Output of podman version:

Version:       1.0.0
Go Version:    go1.11.4
OS/Arch:       linux/amd64

Output of podman info:

host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-1.0.0-1.1.x86_64
    path: /usr/lib/podman/bin/conmon
    version: 'conmon version 1.12.0-dev, commit: '
  Distribution:
    distribution: '"opensuse-tumbleweed-kubic"'
    version: "20190115"
  MemFree: 11926519808
  MemTotal: 16487182336
  OCIRuntime:
    package: runc-1.0.0~rc6-2.1.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 4
  hostname: r2d2
  kernel: 4.20.0-1-default
  os: linux
  rootless: false
  uptime: 7h 35m 10.36s (Approximately 0.29 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: btrfs
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Build Version: 'Btrfs v4.19.1 '
    Library Version: "102"
  ImageStore:
    number: 3
  RunRoot: /var/run/containers/storage
@giuseppe

This comment has been minimized.

Copy link
Member

commented Jan 17, 2019

yes, at the moment we don't honor /etc/containers/libpod.conf for rootless containers, but use directly the version in the user config directory.

Is conmon installed at some path that the default libpod configuration doesn't know about?

@giuseppe giuseppe added the rootless label Jan 17, 2019

@sysrich

This comment has been minimized.

Copy link
Contributor Author

commented Jan 17, 2019

/usr/lib/podman/bin/conmon is in the default libpod configuration documented here: https://github.com/containers/libpod/blob/master/libpod.conf

But the error message implies that podman 1.0.0 is not honoring that default libpod configuration either and looking at a different list of paths somehow.

@vrothberg

This comment has been minimized.

Copy link
Member

commented Jan 17, 2019

/usr/lib/podman/bin/conmon is in the default libpod configuration documented here: https://github.com/containers/libpod/blob/master/libpod.conf

But the error message implies that podman 1.0.0 is not honoring that default libpod configuration either and looking at a different list of paths somehow.

I agree. It seems that when generating ~/.config/containers/libpod.conf the hard-coded defaults are used instead of copying specific values from /etc/containers/libpod.conf.

@vrothberg

This comment has been minimized.

Copy link
Member

commented Jan 17, 2019

Podman should definitely do that for config options. openSUSE is shipping a libpod.conf tailored specifically for openSUSE-specific paths to avoid costly lookups for binaries.

@sysrich

This comment has been minimized.

Copy link
Contributor Author

commented Jan 17, 2019

I agree. It seems that when generating ~/.config/containers/libpod.conf the hard-coded defaults are used instead of copying specific values from /etc/containers/libpod.conf.

I agree, and it seems to me the hard coded defaults are different from the defaults written in the default libpod.conf from https://github.com/containers/libpod/blob/master/libpod.conf

@giuseppe

This comment has been minimized.

Copy link
Member

commented Jan 17, 2019

Podman should definitely do that for config options.

I am not sure about it. We should be careful and cherry-pick only the options that can be inherited safely from the global configuration. Some of them would override the defaults tailored for the rootless mode.

@giuseppe

This comment has been minimized.

Copy link
Member

commented Jan 17, 2019

maybe we should maintain a white list for those that must be inherited from /etc/containers/libpod.conf the first time the rootless configuration is created

@vrothberg

This comment has been minimized.

Copy link
Member

commented Jan 17, 2019

ACK. We shouldn't blindly copy but need to pick the ones that make sense.

@afbjorklund

This comment has been minimized.

Copy link
Contributor

commented Jan 22, 2019

I had the same issue, and had to patch libpod because of it (lack of rootless configuration)

sed -e 's|"/etc/cni|"/usr/local/etc/cni|' -i libpod.conf
sed -e 's|"/etc/cni|"/usr/local/etc/cni|' -i libpod/runtime.go
sed -e 's|/usr/libexec/|/usr/local/lib/|' -i libpod.conf
sed -e 's|/usr/libexec/|/usr/local/lib/|' -i libpod/runtime.go
sed -e 's|/usr/share/|/usr/local/share/|' -i libpod/runtime.go

Otherwise it is not able to find CNI, which means that rootless podman is not able to start up.

Unfortunately it also means it is "dirty", so configuration would be preferred.

Git Commit: "4e0c0ecbc383531cd1b38db9027583974a72070d-dirty"

@rhatdan

This comment has been minimized.

Copy link
Member

commented Jan 22, 2019

We should definitely get the executable paths from libpod.conf

giuseppe added a commit to giuseppe/libpod that referenced this issue Jan 28, 2019

rootless: copy some settings from the global configuration
if some paths are overriden in the global configuration file, be sure
that rootless podman honors them.

Closes: containers#2174

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe

This comment has been minimized.

Copy link
Member

commented Jan 28, 2019

opened a PR here: #2222

@sysrich would it be possible for you to try if it solves the issue you had?

@afbjorklund

This comment has been minimized.

Copy link
Contributor

commented Jan 28, 2019

Would it be possible to add no_pivot_root and cni_config_dir too ?

I think that cgroup_manager is determined automatically (?), so it is OK...

# CGroup Manager - valid values are "systemd" and "cgroupfs"
cgroup_manager = "cgroupfs"

# Whether to use chroot instead of pivot_root in the runtime
no_pivot_root = true

# Directory containing CNI plugin configuration files
cni_config_dir = "/usr/local/etc/cni/net.d/"
@mheon

This comment has been minimized.

Copy link
Collaborator

commented Jan 28, 2019

The systemd cgroup manager doesn't work at all with rootless, so we shouldn't bother sourcing that - it needs to stay unconditionally set to "cgroupfs"

The other two would be nice to have, though

@giuseppe

This comment has been minimized.

Copy link
Member

commented Jan 29, 2019

I can add cni_plugin_dir and no_pivot_root, although rootless containers do not support CNI plugins, so it won't make any difference for now.

cni_config_dir doesn't probably make sense. Even if there will be rootless CNI plugins, the configuration for rootless will be different than from the host

giuseppe added a commit to giuseppe/libpod that referenced this issue Jan 29, 2019

rootless: copy some settings from the global configuration
if some paths are overriden in the global configuration file, be sure
that rootless podman honors them.

Closes: containers#2174

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@afbjorklund

This comment has been minimized.

Copy link
Contributor

commented Jan 29, 2019

The cni_plugin_dir was OK, since it looked in multiple paths (including /usr/local/lib/cni)

I seem to recall some error from CNI, but maybe that was just normal podman and not rootless ?

@afbjorklund

This comment has been minimized.

Copy link
Contributor

commented Jan 29, 2019

Got some other error now: ERRO[0001] slirp4netns failed
Guess I will look into that particular podman error separately.

The other error was (as I suspected) when running as root:
ERRO[0000] error configuring network namespace for container 7dcf3329d7bd6f6372967fcfffaf78bac904246910005bcafc4c4d0260397bca: Missing CNI default network

So no_pivot_root would be enough.

giuseppe added a commit to giuseppe/libpod that referenced this issue Feb 5, 2019

rootless: copy some settings from the global configuration
if some paths are overriden in the global configuration file, be sure
that rootless podman honors them.

Closes: containers#2174

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.