Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for rootless builds #236

Closed
KurtSchluss opened this issue Nov 23, 2018 · 5 comments
Closed

Support for rootless builds #236

KurtSchluss opened this issue Nov 23, 2018 · 5 comments

Comments

@KurtSchluss
Copy link

Hi, IMHO one of the most exciting features of podman is the rootless execution of containers.

I have tried to share /dev/kvm using --mount=type=bind,source=/dev/kvm,destination=/dev/kvm.
This works for init and fetch, but build fails:

user@host ~/coreos % podman --log-level info run -ti --rm --net=host --userns=host --privileged -v $(pwd):/srv --workdir /srv --mount=type=bind,source=/dev/kvm,destination=/dev/kvm quay.io/coreos-assembler/coreos-assembler build
INFO[0000] running as rootless                          
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
INFO[0000] running as rootless                          
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
WARN[0037] Failed to add conmon to cgroupfs sandbox cgroup: mkdir /sys/fs/cgroup/systemd/libpod_parent: permission denied 
Using manifest: /srv/src/config/manifest.yaml
libostree:
 Version: '2018.10'
 Git: c5aaabe9d754d2ada17d5cc006e13105ab5cecc2
 DevelBuild: yes
 Features:
  - libcurl
  - no-http2
  - libsoup
  - gpgme
  - libarchive
  - selinux
  - openssl
  - libmount
  - devel
  - p2p
rpm-ostree:
 Version: '2018.9'
 Git: 0e24944c4ec0a45eb88169a60c96c8b76e86cd61
 Features:
  - compose
  - rust
Previous commit: none
Kickstart checksum: 93de3934c4488ab094a1b8ffce1b00ac1cf8d1c4d7091519202ceb484f2d25fc
Running: rpm-ostree compose tree --repo=/srv/repo --cachedir=/srv/cache --touch-if-changed /srv/tmp/treecompose.changed --unified-core /srv/src/config/manifest.yaml --cache-only --add-metadata-from-json /srv/tmp/build/tmp/commit-metadata-input.json --write-composejson-to /srv/tmp/build/tmp/compose.json
RPM-OSTree Version: 2018.9
No previous commit for fedora/29/x86_64/coreos
Enabled rpm-md repositories: dustymabe-ignition fedora fedora-updates fedora-updates-testing
rpm-md repo 'dustymabe-ignition' (cached); generated: 2018-11-21T22:40:42Z
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'fedora-updates' (cached); generated: 2018-11-23T02:15:41Z
rpm-md repo 'fedora-updates-testing' (cached); generated: 2018-11-23T02:31:21Z
Importing rpm-md... done
Resolving dependencies... done
Installing 388 packages:
  GeoIP-1.6.12-4.fc29.x86_64 (fedora)
...
  zlib-1.2.11-14.fc29.x86_64 (fedora)
Input state hash: 22c7dab4135b85f9d8bea7c3d4e75989a1460b566e1c60f2f5a6972d6f700cdc
Checking out packages... done
⠁ Running pre scripts... libini_config
fuse: device not found, try 'modprobe fuse' first
⠉ Running pre scripts... dbus-common
dbus-common.prein: bwrap: execvp /bin/sh: No such file or directory
fusermount: failed to unmount /tmp/rpmostree-rofiles-fuse.XODVvm: Invalid argument
Running pre scripts... done
error: Running %prein for dbus-common: Executing bwrap(/bin/sh): Child process killed by signal 1
podman --log-level info run -ti --rm --net=host --userns=host --privileged -v  11,15s user 21,99s system 54% cpu 1:00,25 total

Do you see any chance to enable rootless builds?

podman version 0.11.1.1
Arch Linux
@KurtSchluss
Copy link
Author

Adding --mount=type=bind,src=/dev/fuse,dst=/dev/fuse did help.

Build was successful with one error message:

Completed install to disk image: /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
++ pwd
+ /usr/lib/coreos-assembler/gf-anaconda-cleanup /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
+ coreos_gf_run_mount /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
+ coreos_gf_launch /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
+ local src=/srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
+ shift
+ local guestfish
+ guestfish[0]=guestfish
+ guestfish[1]=--listen
+ guestfish[3]=-a
+ guestfish[4]=/srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
++ guestfish --listen -a /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
+ eval 'GUESTFISH_PID=13649; export GUESTFISH_PID'
++ GUESTFISH_PID=13649
++ export GUESTFISH_PID
+ '[' -z 13649 ']'
+ coreos_gf run
+ guestfish --remote -- run
+ local root
++ coreos_gf findfs-label root
++ guestfish --remote -- findfs-label root
+ root=/dev/sda2
+ coreos_gf mount /dev/sda2 /
+ guestfish --remote -- mount /dev/sda2 /
+ local boot
++ coreos_gf findfs-label boot
++ guestfish --remote -- findfs-label boot
+ boot=/dev/sda1
+ coreos_gf mount /dev/sda1 /boot
+ guestfish --remote -- mount /dev/sda1 /boot
++ coreos_gf -findfs-label var
libguestfs: error: findfs_label: findfs exited with status 1: findfs: unable to resolve 'LABEL=var'
+ var=
++ coreos_gf ls /ostree/deploy
++ guestfish --remote -- ls /ostree/deploy
+ stateroot=/ostree/deploy/fedora-coreos
++ coreos_gf ls /ostree/deploy/fedora-coreos/deploy
++ grep -v .origin
++ guestfish --remote -- ls /ostree/deploy/fedora-coreos/deploy
+ deploydir=/ostree/deploy/fedora-coreos/deploy/fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881.0
+ '[' -n '' ']'
+ export stateroot deploydir
+ coreos_gf rm-rf /ostree/deploy/fedora-coreos/deploy/fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881.0/etc/sysconfig/anaconda
+ guestfish --remote -- rm-rf /ostree/deploy/fedora-coreos/deploy/fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881.0/etc/sysconfig/anaconda
+ coreos_gf rm-rf /ostree/deploy/fedora-coreos/deploy/fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881.0/etc/systemd/system/default.target
+ guestfish --remote -- rm-rf /ostree/deploy/fedora-coreos/deploy/fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881.0/etc/systemd/system/default.target
+ coreos_gf rm-rf '/ostree/deploy/fedora-coreos/var/*'
+ guestfish --remote -- rm-rf '/ostree/deploy/fedora-coreos/var/*'
+ coreos_gf_shutdown
+ coreos_gf umount-all
+ guestfish --remote -- umount-all
+ coreos_gf exit
+ guestfish --remote -- exit
+ _coreos_gf_cleanup
+ guestfish --remote -- exit
+ :
++ pwd
++ pwd
+ /usr/lib/coreos-assembler/gf-oemid /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2 /srv/tmp/build/fedora-coreos-29-qemu.qcow2 qemu
++ mktemp -td gf-oemid.XXXXXX
+ tmpd=/tmp/gf-oemid.l7JGf7
+ tmp_dest=/tmp/gf-oemid.l7JGf7/box.img
+ qemu_wrapper=/tmp/gf-oemid.l7JGf7/qemu-wrapper.sh
+ cat
+ chmod +x /tmp/gf-oemid.l7JGf7/qemu-wrapper.sh
+ export LIBGUESTFS_HV=/tmp/gf-oemid.l7JGf7/qemu-wrapper.sh
+ LIBGUESTFS_HV=/tmp/gf-oemid.l7JGf7/qemu-wrapper.sh
+ cp --reflink=auto /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2 /tmp/gf-oemid.l7JGf7/box.img
+ chmod u+w /tmp/gf-oemid.l7JGf7/box.img
+ coreos_gf_run_mount /tmp/gf-oemid.l7JGf7/box.img
+ coreos_gf_launch /tmp/gf-oemid.l7JGf7/box.img
+ local src=/tmp/gf-oemid.l7JGf7/box.img
+ shift
+ local guestfish
+ guestfish[0]=guestfish
+ guestfish[1]=--listen
+ guestfish[3]=-a
+ guestfish[4]=/tmp/gf-oemid.l7JGf7/box.img
++ guestfish --listen -a /tmp/gf-oemid.l7JGf7/box.img
+ eval 'GUESTFISH_PID=13865; export GUESTFISH_PID'
++ GUESTFISH_PID=13865
++ export GUESTFISH_PID
+ '[' -z 13865 ']'
+ coreos_gf run
+ guestfish --remote -- run
+ local root
++ coreos_gf findfs-label root
++ guestfish --remote -- findfs-label root
+ root=/dev/sda2
+ coreos_gf mount /dev/sda2 /
+ guestfish --remote -- mount /dev/sda2 /
+ local boot
++ coreos_gf findfs-label boot
++ guestfish --remote -- findfs-label boot
+ boot=/dev/sda1
+ coreos_gf mount /dev/sda1 /boot
+ guestfish --remote -- mount /dev/sda1 /boot
++ coreos_gf -findfs-label var
libguestfs: error: findfs_label: findfs exited with status 1: findfs: unable to resolve 'LABEL=var'
+ var=
++ coreos_gf ls /ostree/deploy
++ guestfish --remote -- ls /ostree/deploy
+ stateroot=/ostree/deploy/fedora-coreos
++ coreos_gf ls /ostree/deploy/fedora-coreos/deploy
++ guestfish --remote -- ls /ostree/deploy/fedora-coreos/deploy
++ grep -v .origin
+ deploydir=/ostree/deploy/fedora-coreos/deploy/fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881.0
+ '[' -n '' ']'
+ export stateroot deploydir
+ grubcfg_path=/boot/loader/grub.cfg
+ coreos_gf download /boot/loader/grub.cfg /tmp/gf-oemid.l7JGf7/grub.cfg
+ guestfish --remote -- download /boot/loader/grub.cfg /tmp/gf-oemid.l7JGf7/grub.cfg
+ sed -i -e 's, coreos.oem.id=[a-zA-Z0-9]*,,g' /tmp/gf-oemid.l7JGf7/grub.cfg
+ sed -i -e 's,^\(linux16 .*\),\1 coreos.oem.id=qemu,' /tmp/gf-oemid.l7JGf7/grub.cfg
+ coreos_gf upload /tmp/gf-oemid.l7JGf7/grub.cfg /boot/loader/grub.cfg
+ guestfish --remote -- upload /tmp/gf-oemid.l7JGf7/grub.cfg /boot/loader/grub.cfg
++ coreos_gf glob-expand '/boot/loader/entries/ostree-*.conf'
++ guestfish --remote -- glob-expand '/boot/loader/entries/ostree-*.conf'
+ blscfg_path=/boot/loader/entries/ostree-1-fedora-coreos.conf
+ coreos_gf download /boot/loader/entries/ostree-1-fedora-coreos.conf /tmp/gf-oemid.l7JGf7/bls.conf
+ guestfish --remote -- download /boot/loader/entries/ostree-1-fedora-coreos.conf /tmp/gf-oemid.l7JGf7/bls.conf
+ sed -i -e 's, coreos.oem.id=[a-zA-Z0-9]*,,g' /tmp/gf-oemid.l7JGf7/bls.conf
+ sed -i -e 's,^\(options .*\),\1 coreos.oem.id=qemu,' /tmp/gf-oemid.l7JGf7/bls.conf
+ coreos_gf upload /tmp/gf-oemid.l7JGf7/bls.conf /boot/loader/entries/ostree-1-fedora-coreos.conf
+ guestfish --remote -- upload /tmp/gf-oemid.l7JGf7/bls.conf /boot/loader/entries/ostree-1-fedora-coreos.conf
+ coreos_gf_shutdown
+ coreos_gf umount-all
+ guestfish --remote -- umount-all
+ coreos_gf exit
+ guestfish --remote -- exit
+ mv /tmp/gf-oemid.l7JGf7/box.img /srv/tmp/build/fedora-coreos-29-qemu.qcow2
+ rm /tmp/gf-oemid.l7JGf7 -rf
+ _coreos_gf_cleanup
+ guestfish --remote -- exit
+ :
+ set +x
Ignoring non-directory .build-commit
Pruning repo
Total objects: 30887
No unreachable objects
podman --log-level info run -ti --rm --net=host --userns=host --privileged -v  11,14s user 22,20s system 5% cpu 9:55,82 total

Executing run works as well. :)

@jlebon
Copy link
Member

jlebon commented Nov 23, 2018

If you referring to:

++ coreos_gf -findfs-label var
libguestfs: error: findfs_label: findfs exited with status 1: findfs: unable to resolve 'LABEL=var'

You can ignore that. Hmm, it already has 2>/dev/null, so it's printing this on stdout I guess?

Anyway, glad to hear this works!

@KurtSchluss
Copy link
Author

Yes, I did.

Unfortunately there is one exception and I don't know if this is related to the rootless execution, but
re-running build fails:

user@host ~/coreos % podman --log-level info run -ti --rm --net=host --userns=host --privileged -v $(pwd):/srv --workdir /srv --mount=type=bind,source=/dev/kvm,destination=/dev/kvm --mount=type=bind,src=/dev/fuse,dst=/dev/fuse quay.io/coreos-assembler/coreos-assembler build
INFO[0000] running as rootless                          
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
INFO[0000] running as rootless                          
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
WARN[0032] Failed to add conmon to cgroupfs sandbox cgroup: mkdir /sys/fs/cgroup/systemd/libpod_parent: permission denied 
Using manifest: /srv/src/config/manifest.yaml
libostree:
 Version: '2018.10'
 Git: c5aaabe9d754d2ada17d5cc006e13105ab5cecc2
 DevelBuild: yes
 Features:
  - libcurl
  - no-http2
  - libsoup
  - gpgme
  - libarchive
  - selinux
  - openssl
  - libmount
  - devel
  - p2p
rpm-ostree:
 Version: '2018.9'
 Git: 0e24944c4ec0a45eb88169a60c96c8b76e86cd61
 Features:
  - compose
  - rust
Previous build: 29
Previous commit: fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881
Kickstart checksum: 93de3934c4488ab094a1b8ffce1b00ac1cf8d1c4d7091519202ceb484f2d25fc
Running: rpm-ostree compose tree --repo=/srv/repo --cachedir=/srv/cache --touch-if-changed /srv/tmp/treecompose.changed --unified-core /srv/src/config/manifest.yaml --cache-only --add-metadata-from-json /srv/tmp/build/tmp/commit-metadata-input.json --write-composejson-to /srv/tmp/build/tmp/compose.json
RPM-OSTree Version: 2018.9
Previous commit: fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881
error: fsetxattr: Operation not permitted
podman --log-level info run -ti --rm --net=host --userns=host --privileged -v  11,22s user 21,23s system 84% cpu 38,480 total

After a clean, build succeeds again.

@jlebon
Copy link
Member

jlebon commented Nov 23, 2018

Hmm, not sure. Though you should be able to use the supermin backend instead (#190). Should be able to just drop the --privileged flag and it'll automatically fallback to that, or otherwise use FORCE_UNPRIVILEGED. But note this requires an up to date rpm-ostree: #194 (comment)

@KurtSchluss
Copy link
Author

Thank you very much for that hint, dropping --privileged works like a charm!
Also, re-running build causes no issue anymore.
Only thing I had to do was wiping the directory and starting from scratch.

For the record, here is the alias I have been using:
alias coreos-assembler='podman run -ti --rm --net=host --userns=host -v $(pwd):/srv --workdir /srv --mount=type=bind,source=/dev/kvm,destination=/dev/kvm --mount=type=bind,src=/dev/fuse,dst=/dev/fuse quay.io/coreos-assembler/coreos-assembler'

@jlebon jlebon mentioned this issue Dec 17, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jlebon @KurtSchluss