yum fails to run in a container built/launched with rootless buildah/podman

[junwang123]

Description
The build of CentOS 7 container base image using this approach works with docker, but not with rootless buildah/podman due to a yum failure in chroot.sh. For the detailed log, refer to centos7-buildah-podman.log on issue #362 of base-images-docker repo.

Steps to reproduce the issue:
On a system running CentOS 7.6.1810, and with buildah/podman and vbatts's shadow-utils-newxidmap installed,

$git clone https://github.com/GoogleContainerTools/base-images-docker.git
$cd base-images-docker/centos
$buildah bud -f Dockerfile.build -t builder .
$podman run --privileged -v $(pwd):/workspace builder /build.sh

Describe the results you received:
In running yum -y -q --releasever=7 install yum centos-release in chroot /target ./chroot.sh in build.sh,

error: Failed to initialize NSS library
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

cannot import name ts

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.5 (default, Apr 9 2019, 14:30:50)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

If you cannot solve this problem yourself, please go to
the yum faq at:
http://yum.baseurl.org/wiki/Faq

Describe the results you expected:
yum install succeeds and chroot.sh proceeds to build a base container image.

Output of rpm -q buildah or apt list buildah:

buildah-1.7.1-2.git93747b1.el7.centos.x86_64
podman-0.12.1.2-2.git9551f6b.el7.centos.x86_64

Output of buildah version:

buildah version 1.7.1 (image-spec 1.0.0, runtime-spec 1.0.0)

Output of podman version if reporting a podman build issue:

podman version 0.12.1.2

Output of cat /etc/*release:

CentOS Linux release 7.6.1810 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.6.1810 (Core)
CentOS Linux release 7.6.1810 (Core)

Output of uname -a:

Linux osbox76 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# mountopt specifies extra mount options used when mounting the thin devices.
# mountopt = ""

# use_deferred_removal Marking device for deferred removal
# use_deferred_removal = "True"

# use_deferred_deletion Marking device for deferred deletion
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

[TomSweeneyRedHat]

Thanks for the issue report @junwang123
@giuseppe can you take a look please?

[rhatdan]

We really need to repeat this on a little more up2date versions of podman and buildah. Decent chance that these could be fixed in the latest releases.

[junwang123]

yum fails because /dev/urandom doesn't work inside the /target chroot environment even though mount --bind /dev/ /target/dev/ is done outside the chroot.

225 mprotect(0x7f5ade7a6000, 8192, PROT_READ) = 0
225 munmap(0x7f5ae350f000, 9114) = 0
225 open("/dev/urandom", O_RDONLY) = 6
225 read(6, "", 110) = 0
225 close(6) = 0
225 open("/dev/urandom", O_RDONLY) = 6
225 read(6, "", 55) = 0
225 close(6) = 0
225 open("/proc/sys/crypto/fips_enabled", O_RDONLY) = 6
225 fstat(6, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
225 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5ae3511000
225 read(6, "0\n", 1024) = 2
225 close(6) = 0
225 munmap(0x7f5ae3511000, 4096) = 0
225 write(2, "error: ", 7) = 7
225 write(2, "Failed to initialize NSS library"..., 33) = 33
225 close(5) = 0

In a privileged container launched by docker (running on macOS), reading from /dev/urandom inside a chroot environment works fine.

230 mprotect(0x7faca03da000, 8192, PROT_READ) = 0
230 munmap(0x7faca5143000, 9114) = 0
230 open("/dev/urandom", O_RDONLY) = 6
230 read(6, "<ew\0030}g\31\n'\342\5PQ\270\305o\0\273>LS\0310PjX/\242\311\337\230"..., 110) = 110
230 close(6) = 0
230 sysinfo({uptime=141143, loads=[49632, 19232, 8832], totalram=2095869952, freeram=224768000, sharedram=847872, bufferram=125456384, totalswap=1073737728, freeswap=1071833088, procs=478, totalhigh=0, freehigh=0, mem_unit=1}) = 0
230 gettimeofday({1560181724, 45869}, NULL) = 0
230 uname({sysname="Linux", nodename="5c18a3ba3ac1", ...}) = 0
230 stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...}) = 0
230 open("/dev/urandom", O_RDONLY) = 6
230 read(6, "\324\4I?\306\27p\271\241\251\353\211 \4\310g\233\345gt8=\310=k\200\26\346\371`\250o"..., 1024) = 1024
230 close(6) = 0
230 gettimeofday({1560181724, 47340}, NULL) = 0
230 stat("/etc/passwd", {st_mode=S_IFREG|0644, st_size=513, ...}) = 0
230 open("/etc/passwd", O_RDONLY) = 6

[junwang123]

We really need to repeat this on a little more up2date versions of podman and buildah. Decent chance that these could be fixed in the latest releases.
@rhatdan
I did try a newer buildah, but its rootless run doesn't work well on CentOS 7.6 with systemd-219. There is discussion online about getting rid of the error, but it seems to be with newer systemd (systemd-233?) and systemd-pam.

$ rpm -q buildah
buildah-1.8.1-1.git57b4288.el7.x86_64
$ buildah bud -f Dockerfile.build -t builder .
mkdir /var/run/containers/storage: permission denied
ERRO[0000] exit status 1
$ buildah images
mkdir /var/run/containers/storage: permission denied
ERRO[0000] exit status 1

[rhatdan]

This was fixed in even newer versions of buildah, I believe.

[junwang123]

@rhatdan
Thanks for the help. I tried packages built at CentOS Community Build Service.

buildah 1.9 appears to work fine.

With podman 1.2,

[osboxes@osbox76 centos]$ podman run --rm -it --privileged -v $(pwd):/workspace builder /bin/bash
Error: container create failed: cannot specify gid= mount options for unmapped gid in rootless containers
: internal libpod error
[osboxes@osbox76 centos]$ rpm -qa podman
podman-1.2-2.git3bd528e.el7.x86_64
[osboxes@osbox76 ~]$ rpm -qa runc
runc-1.0.0-60.dev.git2abd837.el7.x86_64

The attempt to upgrade podman to 1.4.0 failed due to epoch number of the runc package.

sudo yum localinstall podman-1.4.0-1.el7.x86_64.rpm podman-manpages-1.4.0-1.el7.noarch.rpm
... snip ...

--> Finished Dependency Resolution
Error: Package: podman-1.4.0-1.el7.x86_64 (/podman-1.4.0-1.el7.x86_64)
Requires: runc >= 2:1.0.0-57
Installed: runc-1.0.0-60.dev.git2abd837.el7.x86_64 (installed)
runc = 1.0.0-60.dev.git2abd837.el7
Available: runc-1.0.0-52.dev.git70ca035.el7_5.x86_64 (extras)
runc = 1.0.0-52.dev.git70ca035.el7_5
Available: runc-1.0.0-54.dev.git2abd837.el7.x86_64 (extras)
runc = 1.0.0-54.dev.git2abd837.el7
Available: runc-1.0.0-57.dev.git2abd837.el7.centos.x86_64 (extras)
runc = 1.0.0-57.dev.git2abd837.el7.centos
Available: runc-1.0.0-59.dev.git2abd837.el7.centos.x86_64 (extras)
runc = 1.0.0-59.dev.git2abd837.el7.centos
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest

[rhatdan]

Awesome. We should be getting these newer versions in in the next release.

[junwang123]

@rhatdan , buildah 1.9 works just for building the builder image, but neither podman nor buildah was able to successfully run to the point where the originally-reported yum in chroot issue can be tested. Could you please keep the issue open to continue the investigation? Thanks.

[osboxes@osbox76 centos]$ buildah run -v $(pwd):/workspace builder /build.sh
error reading build container "builder": error reading build container: container not known
ERRO[0000] exit status 1
[osboxes@osbox76 centos]$ podman run --privileged -v $(pwd):/workspace builder /build.sh
Error: container create failed: cannot specify gid= mount options for unmapped gid in rootless containers
: internal libpod error

[junwang123]

podman issue #3270

[junwang123]

@rhatdan, Using the --cap-add=CAP_SYS_ADMIN option on buildah run, I was able to work around the podman problem and to reproduce the original yum failure in chroot problem. --cap-add=CAP_SYS_ADMIN --cap-add=CAP_SYS_CHROOT and --cap-add=CAP_SYS_ADMIN,CAP_SYS_CHROOT options on buildah run were also tried.

The original yum failure in chroot problem were still there with all of these options. These are with the buildah 1.8.2 package in RHEL 7.7 beta. Refer to the full log at buildah-run-with-cap_system_admin.log
.

[osboxes@osbox76 centos]$ rpm -q podman buildah runc
podman-1.3.2-1.git14fdcd0.el7.x86_64
buildah-1.8.2-2.gite23314b.el7.x86_64
runc-1.0.0-59.dev.git2abd837.el7.x86_64

+ chroot /target ./chroot.sh
+ yum -y -q --releasever=7 install yum centos-release
error: Failed to initialize NSS library
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

cannot import name ts

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.5 (default, Apr 9 2019, 14:30:50)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

If you cannot solve this problem yourself, please go to
the yum faq at:
http://yum.baseurl.org/wiki/Faq

[junwang123]

mknod doesn't work in containers launched with buildah run even with many capabilities, including CAP_MKNOD, enabled, refer to mknod.strace.log.

buildah run --cap-add=CAP_SYS_CHROOT --cap-add=CAP_SYS_ADMIN --cap-add=CAP_SYS_PTRACE --cap-add=CAP_MKNOD --cap-add=CAP_SETGID --cap-add=CAP_SETPCAP --cap-add=CAP_SETUID $(buildah from builder) /bin/bash

• mkdir -p /target/dev
• mknod /target/dev/random c 1 8
  mknod: '/target/dev/random': Operation not permitted
• mknod /target/dev/urandom c 1 9
  mknod: '/target/dev/urandom': Operation not permitted

[giuseppe]

an unprivileged user (rootless) has not enough privileges for using mknod, the kernel blocks that. It doesn't matter how many capabilities are left in the user namespace, it won't still be able to use it.

[rhatdan]

Correct, if you have to create device nodes, then you have to run as root.

[junwang123]

Thanks Giuseppe and Daniel.
Would we be able to use mknod in a container launched with rootless podman run --privileged?

In addtion, in a container launched with a rootless run, i.e. with buildah run --cap-add and with podman run --privileged respectively, is bind mounting /dev to /chroot_directory/dev expected to allow /dev/urandom to function in the chroot environment? I tried mknod only because bind mount doesn't work in that case.

mount --bind /dev/ /chroot_directory/dev/
chroot /chroot_directory od -N 3 /dev/urandom

[giuseppe]

Would we be able to use mknod in a container launched with rootless podman run --privileged?

no, there is no way to get mknod working with rootless containers. --privileged just means create a user namespace but not drop any of the capabilities I've got.

Bind mounting a device from the host should work fine:

$ podman run --rm -ti -v /dev/urandom:/tmp/urandom alpine dd if=/tmp/urandom of=/dev/zero count=1 bs=1
1+0 records in
1+0 records out

it will show up in the container owned by the nobody user since root is not mapped in the rootless container, but it is still fine as it is world writeable/readable so that will still hold in the user namespace.

[rhatdan]

@giuseppe is somewhat correct on --privileged. But I would like to clarify it a little.

--privileged in a rootless container means, do not further restrict the container from a security point of view, then the program launching podman. From a Linux Capability point of view, it says do not modify the current NAMESPACED capabilities. Since most likely the user process that is launching the container does not have any REAL capabilities. A Even though you might have a Namespaced capability of MKNOD, it does not mean it allows you to create Device nodes, since the process would also need a REAL Capability of MKNOD.

[junwang123]

@giuseppe, Bind mounting the /dev/urandom device from the host with the podman -v option does make the /dev/urandom and yum/rpm commands work in chroot environments launched inside the container. My original issue is now resolved. Thanks a lot.

@rhatdan, I see, so the effects of those options are subject to capability inheritance limits from the container launcher user process. Thanks for the explanation.