podman does not adhere to path set with LIBEXECDIR (e.g. for catatonit) #1110

dvzrv · 2022-08-03T07:40:18Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Podman only accepts executables in /usr/libexec/podman (even if LIBEXECDIR is set to something else). This conflicts with OSes, that do not use /usr/libexec (e.g. Arch Linux) and package catatonit in /usr/lib/podman/catatonit.

Steps to reproduce the issue:

make EXTRA_LDFLAGS='-s -w -linkmode=external' -C $pkgbase & make install install.completions DESTDIR="$pkgdir" PREFIX=/usr LIBEXECDIR=/usr/lib -C $pkgbase (or install podman 4.1.1-2 on Arch Linux, which pulls in catatonit symlinked to /usr/lib/podman/catatonit)
Run podman run --init docker.io/library/busybox

Describe the results you received:

"Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory"

(see downstream bug report https://bugs.archlinux.org/task/75493)

Describe the results you expected:

Podman's build system can be configured with LIBEXECDIR and podman afterwards adheres to that configured directory.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.18.3
Git Commit:   f73d8f8875c2be7cd2049094c29aff90b1150241
Built:        Fri Jun 24 09:33:56 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.3-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.3, commit: ab52a597278b20173440140cd810dc9fa8785c93'
  cpuUtilization:
    idlePercent: 99.04
    systemPercent: 0.41
    userPercent: 0.55
  cpus: 24
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: hmbx
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 4096
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 4096
  kernel: 5.18.12-hardened1-1-hardened
  linkmode: dynamic
  logDriver: journald
  memFree: 1003122688
  memTotal: 67342626816
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.5-1
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 21712936960
  swapTotal: 22447202304
  uptime: 312h 9m 12.15s (Approximately 13.00 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/dave/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /home/dave/.local/share/containers/storage
  graphRootAllocated: 999662751744
  graphRootUsed: 615612772352
  graphStatus:
    Build Version: Btrfs v5.18.1
    Library Version: "102"
  imageCopyTmpDir: /run/user/1000
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/dave/.local/share/containers/storage/volumes
version:
  APIVersion: 4.1.1
  Built: 1656056036
  BuiltTime: Fri Jun 24 09:33:56 2022
  GitCommit: f73d8f8875c2be7cd2049094c29aff90b1150241
  GoVersion: go1.18.3
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

Name            : podman
Version         : 4.1.1-2
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/podman
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : catatonit  conmon  containers-common  crun  iptables  libdevmapper.so=1.02-64  libgpgme.so=11-64  libseccomp.so=2-64  slirp4netns
Optional Deps   : apparmor: for AppArmor support [installed]
                  btrfs-progs: support btrfs backend devices [installed]
                  netavark: for a new container-network-stack implementation [installed]
                  podman-compose: for docker-compose compatibility [installed]
                  podman-docker: for Docker-compatible CLI
Required By     : cockpit-podman  podman-compose
Optional For    : python-pytest-testinfra
Conflicts With  : None
Replaces        : None
Installed Size  : 63.27 MiB
Packager        : David Runge <dvzrv@archlinux.org>
Build Date      : 2022-06-24T09:33:56 CEST
Install Date    : 2022-06-24T09:38:16 CEST
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes (not relevant for this ticket)

Additional environment details (AWS, VirtualBox, physical, etc.):

Locations in which hardcoding in the codebase takes place:
https://github.com/containers/podman/blob/c09457e34a429622475e27fe68e17effe47fe0c3/pkg/rootless/rootless_linux.c#L137
https://github.com/containers/podman/blob/c09457e34a429622475e27fe68e17effe47fe0c3/test/e2e/run_test.go#L319

Locations in containers/common that hardcode /usr/libexec/podman:

common/pkg/config/containers.conf

Line 149 in ab553b9

#init_path = "/usr/libexec/podman/catatonit"

common/pkg/config/containers.conf

Line 360 in ab553b9

# "/usr/libexec/podman/conmon",

common/pkg/config/default.go

Line 53 in ab553b9

DefaultInitPath = "/usr/libexec/podman/catatonit"

There are probably more, but it's hard to estimate with the components being a bit of a moving target.

The text was updated successfully, but these errors were encountered:

dvzrv · 2022-08-03T08:18:38Z

This seems to only work if the default configuration (which has init_path commented!) has the init_path set to /usr/lib/podman/catatonit (not commented!).

I was not able to figure out based on the codebase where exactly the implicit default of /usr/libexec/podman/catatonit is set. Changing the above mentioned code sections did not fix this issue.

How does podman assemble the configuration default, if init_path is commented?

dvzrv · 2022-08-03T08:31:11Z

FWIW, I've corrected a few typos in my initial report.

afbjorklund · 2022-08-03T09:59:09Z

Historically it always needed patching (code), even if there were some attempts to make it configurable

rhatdan · 2022-08-03T10:38:18Z

init_path default is set in containers/common/pkg/config/default.go
https://github.com/containers/common/blob/main/pkg/config/default.go#L53

Luap99 · 2022-08-03T14:32:50Z

IMO there is no reason to only use one path for the init binary. We already have the helper_binaries_dir field to specify a list of directories. I think we should look there as well.

Foxboron · 2022-08-03T19:29:21Z

Talking about containers/common Do you want a list of entries for InitPath like there is for ConmonPath? Or should there just be a gracefull fallback if the path doesn't exist?

rhatdan · 2022-08-04T10:30:25Z

@dvzrv Interested in opening a PR to change defaults.go to specify the default location of inits_path based on searching helper_binaries_dir=["/usr/libexec/podman", ...]

dvzrv · 2022-08-04T11:08:50Z

I'm afraid my go skills are not really sufficient to do this in a reasonable time frame. Potentially @Foxboron might be interested though!

afbjorklund · 2022-08-04T12:53:53Z

I think I used sed 🙂

And a patch to remove the array, but if you want it runtime-configurable...

dvzrv · 2022-08-04T13:07:30Z

I think I used sed

Sure, this can be worked around in downstreams, but doing this for all sorts of packages is not feasible and does not scale at all (especially if you maintain hundreds of packages) :)
This is an easy enough target to get fixed in this upstream to be able to then run well on many downstreams without any modifications at all.

Luap99 · 2022-08-04T13:11:11Z

We should keep init_path for backwards compat but if it does not exists we should lookup catatonit in helper_binaries_dir.
A extra helper_binaries_dir can also be set from the podman Makefile via HELPER_BINARIES_DIR=...

Some Linux distributions does not install files into `/usr/libexec` and would fail to lookup the init binary. This change first checks if the file exists in the default location, else we look for the binary in helper_binaries_dir as they should encompass all the possible installation directories. Fixes containers#1110 Signed-off-by: Morten Linderud <morten@linderud.pw>

unknowndevQwQ · 2022-10-13T07:37:14Z

Should we consider addressing it afterwards?
#809

dvzrv · 2023-02-11T11:38:20Z

Hi! What's the status on this?

rhatdan · 2023-02-14T05:10:47Z

Interested in opening a PR to fix the issue?

dvzrv · 2023-07-21T12:30:44Z

Interested in opening a PR to fix the issue?

AFAIK, there's already #1115 by @Foxboron attempting to fix it.

Meanwhile I still have to patch this downstream and it would be very nice if this wasn't needed!

Luap99 · 2023-10-17T12:27:57Z

I tackle this one.

Forcing a single upstream default for the init path is bad as some distro use differnt install locations for various reasons. To fix this use the existing helper_binaries_dir field to lookup in all directories. To keep backwards compatibility we keep using the old default and both Containers.InitPath and Engine.InitPath. Yes that is right somehow we ended up with the same config field under the contianers and engine section and they are both used in podman! Thus we need to keep supporting both, only the field under the container section was documnted and it will now be marked as deprecated. To make the docs more clear also document what binaries are currently looked up in helper_binaries_dir. Note this needs further integration in podman. Fixes containers#1110 Signed-off-by: Paul Holzinger <pholzing@redhat.com>

Forcing a single upstream default for the init path is bad as some distro use different install locations for various reasons. To fix this use the existing helper_binaries_dir field to lookup in all directories. To keep backwards compatibility we keep using the old default and both Containers.InitPath and Engine.InitPath. Yes that is right, somehow we ended up with the same config field under the containers and engine section and they are both used in podman! Thus we need to keep supporting both, only the field under the container section was documented and it will now be marked as deprecated. To make the docs more clear also document what binaries are currently looked up in helper_binaries_dir. Note this needs further integration in podman. Fixes containers#1110 Signed-off-by: Paul Holzinger <pholzing@redhat.com>

Use the new FindInitBinary() function to lookup the init binary, this allows the use of helper_binaries_dir in contianers.conf[1] [NO NEW TESTS NEEDED] [1] containers/common#1110 Signed-off-by: Paul Holzinger <pholzing@redhat.com>

Forcing a single upstream default for the init path is bad as some distro use different install locations for various reasons. To fix this use the existing helper_binaries_dir field to lookup in all directories. To keep backwards compatibility we keep using the old default and both Containers.InitPath and Engine.InitPath. Yes that is right, somehow we ended up with the same config field under the containers and engine section and they are both used in podman! Thus we need to keep supporting both, only the field under the container section was documented and now recommends the use of helper_binaries_dir. To make the docs more clear also document what binaries are currently looked up in helper_binaries_dir. Note this needs further integration in podman. Fixes containers#1110 Signed-off-by: Paul Holzinger <pholzing@redhat.com>

Use the new FindInitBinary() function to lookup the init binary, this allows the use of helper_binaries_dir in contianers.conf[1] [NO NEW TESTS NEEDED] [1] containers/common#1110 Signed-off-by: Paul Holzinger <pholzing@redhat.com>

Foxboron · 2023-10-18T13:33:18Z

@Luap99 Thank you!

openshift-ci bot added the kind/bug label Aug 3, 2022

rhatdan transferred this issue from containers/podman Aug 4, 2022

rhatdan added the good first issue Good for newcomers label Aug 4, 2022

Foxboron mentioned this issue Aug 7, 2022

default: look up init binary in helper_binaries_dir #1115

Closed

hswong3i mentioned this issue Sep 6, 2022

Error Container-init binary not found on the host alvistack/ansible-role-podman#10

Closed

mgoltzsche mentioned this issue Apr 5, 2023

container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory mgoltzsche/podman-static#54

Closed

Luap99 self-assigned this Oct 17, 2023

Luap99 mentioned this issue Oct 17, 2023

pkg/config: lookup InitPath in HelperBinariesDir #1698

Merged

Luap99 mentioned this issue Oct 17, 2023

use FindInitBinary() for init binary containers/podman#20383

Merged

openshift-ci bot closed this as completed in #1698 Oct 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podman does not adhere to path set with LIBEXECDIR (e.g. for catatonit) #1110

podman does not adhere to path set with LIBEXECDIR (e.g. for catatonit) #1110

dvzrv commented Aug 3, 2022 •

edited

Loading

dvzrv commented Aug 3, 2022 •

edited

Loading

dvzrv commented Aug 3, 2022

afbjorklund commented Aug 3, 2022

rhatdan commented Aug 3, 2022

Luap99 commented Aug 3, 2022

Foxboron commented Aug 3, 2022

rhatdan commented Aug 4, 2022

dvzrv commented Aug 4, 2022

afbjorklund commented Aug 4, 2022 •

edited

Loading

dvzrv commented Aug 4, 2022

Luap99 commented Aug 4, 2022

unknowndevQwQ commented Oct 13, 2022

dvzrv commented Feb 11, 2023

rhatdan commented Feb 14, 2023

dvzrv commented Jul 21, 2023 •

edited

Loading

Luap99 commented Oct 17, 2023

Foxboron commented Oct 18, 2023

podman does not adhere to path set with LIBEXECDIR (e.g. for catatonit) #1110

podman does not adhere to path set with LIBEXECDIR (e.g. for catatonit) #1110

Comments

dvzrv commented Aug 3, 2022 • edited Loading

dvzrv commented Aug 3, 2022 • edited Loading

dvzrv commented Aug 3, 2022

afbjorklund commented Aug 3, 2022

rhatdan commented Aug 3, 2022

Luap99 commented Aug 3, 2022

Foxboron commented Aug 3, 2022

rhatdan commented Aug 4, 2022

dvzrv commented Aug 4, 2022

afbjorklund commented Aug 4, 2022 • edited Loading

dvzrv commented Aug 4, 2022

Luap99 commented Aug 4, 2022

unknowndevQwQ commented Oct 13, 2022

dvzrv commented Feb 11, 2023

rhatdan commented Feb 14, 2023

dvzrv commented Jul 21, 2023 • edited Loading

Luap99 commented Oct 17, 2023

Foxboron commented Oct 18, 2023

dvzrv commented Aug 3, 2022 •

edited

Loading

dvzrv commented Aug 3, 2022 •

edited

Loading

afbjorklund commented Aug 4, 2022 •

edited

Loading

dvzrv commented Jul 21, 2023 •

edited

Loading