podman support

[hoshsadiq]

Could we get podman support? I noticed in 2 other issues you mentioned that you're waiting for deb packages to be released. Not sure when they were released, but instructions on installation are available for Debian based distros.

[eine]

Podman is also the default on latest Fedora, and installing docker can be not straightforward.

[mviereck]

I am interested in supporting podman, too.
I could straightforward install it now on Debian testing from the Debian repository. For this I was waiting.

A first test run just typing podman gives an error:

$ podman
cannot clone: Operation not permitted
user namespaces are not enabled in /proc/sys/kernel/unprivileged_userns_clone
Error: could not get runtime: cannot re-exec process

I've included an experimental x11docker option --podman That just runs podman instead of docker.
A first test run with sudo x11docker --podman -d x11docker/fvwm gives me:

$ sudo ~/git/x11docker/x11docker --podman -d x11docker/fvwm
WARN[0000] unable to find /etc/containers/registries.conf. some podman (image shortnames) commands may be limited 
WARN[0000] unable to find /etc/containers/registries.conf. some podman (image shortnames) commands may be limited 
WARN[0000] unable to find /etc/containers/registries.conf. some podman (image shortnames) commands may be limited 
Image x11docker/fvwm not found locally.
Do you want to pull it from docker hub?
(timeout after 60s assuming no) [Y|n]
x11docker note: Pulling image 'x11docker/fvwm' from docker hub

Error: error pulling image "x11docker/fvwm": unable to pull x11docker/fvwm: image name provided is a short name and no search registries are defined in the registries config file.

So it seems podman needs some setup beforehand.

[hoshsadiq]

So I've setup a soft link for docker to become podman (which is apparently a relatively common pattern, as there's a podman-docker package for example), based on this I tried running x11docker (which picked up the link and thus ran it using podman). I tried running kodi, but no success. I will try with --podman changes later.

So it seems podman needs some setup beforehand.

I had some issues running it as well. Can you show me the output of the following commands?:

podman pull x11docker/fvwm
podman unshare cat /proc/self/uid_map

Also, do you have the packages uidmap and slirp4netns installed? If you get some along the lines of there might not be enough IDs available in the namespace or to that effect, simply ensure uidmap and slirp4netns is installed, then run podman system migrate. After running this command the above unshare command should output an additional line compared to before.

[mviereck]

do you have the packages uidmap and slirp4netns installed?

Yes.

$ podman pull x11docker/fvwm
cannot clone: Operation not permitted
user namespaces are not enabled in /proc/sys/kernel/unprivileged_userns_clone
Error: could not get runtime: cannot re-exec process

$ podman unshare cat /proc/self/uid_map
cannot clone: Operation not permitted
user namespaces are not enabled in /proc/sys/kernel/unprivileged_userns_clone
Error: could not get runtime: cannot re-exec process

$ podman system migrate
cannot clone: Operation not permitted
user namespaces are not enabled in /proc/sys/kernel/unprivileged_userns_clone
Error: could not get runtime: cannot re-exec process

$ sudo podman system migrate
$

$ podman unshare cat /proc/self/uid_map
cannot clone: Operation not permitted
user namespaces are not enabled in /proc/sys/kernel/unprivileged_userns_clone
Error: could not get runtime: cannot re-exec process

$ sudo podman unshare cat /proc/self/uid_map
Error: please use unshare with rootless

$ cat /proc/sys/kernel/unprivileged_userns_clone
0

$ sudo podman pull x11docker/fvwm
Error: error pulling image "x11docker/fvwm": unable to pull x11docker/fvwm: image name provided is a short name and no search registries are defined in the registries config file.

[jd666]

This is what I use to get a working container environment:

rel=10;    ## 10, Testing, Unstable
kr=/usr/share/keyrings/libcontainers-deb-${rel}-keyring.gpg
curl -s -o ${kr}-tmp https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_$rel/Release.key
rm -f $kr; gpg --no-default-keyring --keyring=$kr --import ${kr}-tmp && rm ${kr}-tmp
echo "deb [signed-by=${kr}] https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_$rel/ /" > /etc/apt/sources.list.d/libcontainers-deb-$rel.list
apt update && apt install podman buildah crun slirp4netns varlink

The error message indicates that somehow during the setup a kernel parameter wasn't set. You can run sudo sysctl -w kernel.unprivileged_userns_clone=1 to work around it. This disables a bit of "hardening" that Debian patches into their distribution kernel. If you're not running such a kernel, it will fail and not do anything, as such a setting doesn't even exist in the mainline Linux kernel. If you were running such a patched kernel, all it would do is disable the functionality of that patch, and let your kernel work like every other kernel, allowing unprivileged users to use unshare -U. ((Please note that this is weakening the security as it allows non-root users to use kernel namespaces -- but that is exactly what unprivileged containers do..))

[mviereck]

sudo sysctl -w kernel.unprivileged_userns_clone=1

Thank you! That was a missing key, and thanks for the further explanation of it.

Another missing point was etc/containers/registries.conf (partially taken from https://podman.io/getting-started/installation.html):

$ cat /etc/containers/registries.conf 
[registries.search]
registries = ['docker.io']

But pulling from docker hub fails:

$ podman pull x11docker/fvwm
Trying to pull docker.io/x11docker/fvwm...
Getting image source signatures
Copying blob 77025076aa24 done
Copying blob aad63a933944 done
  read tcp 192.168.178.46:44284->104.18.123.25:443: read: connection reset by peer
Error: error pulling image "x11docker/fvwm": unable to pull x11docker/fvwm: 1 error occurred:
	* Error writing blob: error storing blob to file "/var/tmp/storage448539135/2": read tcp 192.168.178.46:44284->104.18.123.25:443: read: connection reset by peer

It might be an issue due to my low internet bandwith and docker hub just interrupting a slow download.

Using the local docker images failes, too (https://developers.redhat.com/blog/2019/02/21/podman-and-buildah-for-docker-users/):

$ podman pull  docker-daemon:x11docker/fvwm
Error: error pulling image "docker-daemon:x11docker/fvwm": unable to pull docker-daemon:x11docker/fvwm: error getting default registries to try: invalid reference format

Maybe the local docker images need an entry in /etc/containers/registries.conf , too.

[mviereck]

First success!

The podman pull command fails for unknown reasons in podman itself, it's not to blame docker hub (that also sometimes interrupts my downloads). I assume podman tries to store the image in a root-owned folder.

Running everything with sudo fixes those issues for now. To run podman unprivileged, some better setup should be done by the Debian package.

Once I had a podman image x11docker/fvwm, x11docker --podman failed with an apparmor error message. I could fix it with --security-opt apparmor=unconfined. Apparmor is running on my Debian 11/bullseye. Maybe podman does not provide a valid profile.

This one works so far:

sudo x11docker --podman --desktop -- --security-opt apparmor=unconfined -- x11docker/fvwm

Interestingly, this works so far without sudo sysctl -w kernel.unprivileged_userns_clone=1. x11docker sets --userns=host to allow host file sharing; that might be the reason.

Edit:
Currently it makes sense to me to wait some weeks before I try to fix the current issues. The podman package in Debian testing is quite fresh and will likely get some fixes soon.

[adreyer666]

I had x11docker --podman working and then updated today..
Now I am getting errors and it looks like dockeris hardceded somewhere in a dockerrc:

$ x11docker --podman x11docker/check
x11docker note: Option --podman: experimental only.

x11docker note: Using X server option --xpra

x11docker note: Option --xpra: If you encounter issues with xpra, 
  you can try --nxagent instead.

/home/adreyer/.cache/x11docker/x11docker-check-17277368301/dockerrc: line 542: docker: command not found

x11docker ERROR: dockerrc(): Did not receive PID of PID1 in container. 
  Maybe the container immediately stopped forunknown reasons. 
  Output of "docker ps | grep x11docker": 

...

I can send the full verbose log if needed.

[mviereck]

I found the hardcoded docker command, it was part of the error message. Is fixed now.
But this could not cause the startup failure.

Can you please provide the full log at www.pastebin.com?

[adreyer666]

https://pastebin.com/e9y1738u

[mviereck]

I found the messages of interest:

==> /home/adreyer/.cache/x11docker/x11docker-check-16958756445/message.log <==
DEBUGNOTE[09:29:27,238]: storeinfo(): containerid=b06a56814aabdb715e903d1babd06255a1dfb4973200ac84f52b9070230946ad
 
==> /home/adreyer/.cache/x11docker/x11docker-check-16958756445/share/container.log <==
Error: container b06a56814aabdb715e903d1babd06255a1dfb4973200ac84f52b9070230946ad does not exist in database: no such container
 
==> /home/adreyer/.cache/x11docker/x11docker-check-16958756445/message.log <==
DEBUGNOTE[09:29:27,805]: dockerrc: Container is up and running.
DEBUGNOTE[09:29:27,823]: storepid(): Stored pid '1090534' of 'dockerlogs': 1090534 pts/13   00:00:00 podman
 
==> /home/adreyer/.cache/x11docker/x11docker-check-16958756445/share/stderr <==
Error: no container with name or ID b06a56814aabdb715e903d1babd06255a1dfb4973200ac84f52b9070230946ad found: no such container
 
==> /home/adreyer/.cache/x11docker/x11docker-check-16958756445/share/container.log <==
Error: error inspecting object: no such object: "x11docker_X113_x11docker-check_16958756445"
 
==> /home/adreyer/.cache/x11docker/x11docker-check-16958756445/message.log <==
DEBUGNOTE[09:29:27,858]: dockerrc: 1. check for PID 1: 
 
==> /home/adreyer/.cache/x11docker/x11docker-check-16958756445/share/container.log <==
Error: error inspecting object: no such object: "x11docker_X113_x11docker-check_16958756445"

x11docker receives a container id from podman. But asking podman for the container with this id results in

Error: container b06a56814aabdb715e903d1babd06255a1dfb4973200ac84f52b9070230946ad does not exist in database: no such container

The related code in dockerrc:

   509	[ "$Containerid" ] || {
   510	    error "Startup of docker failed. Did not receive a container ID.
   511	    
   512	  Last lines of container log:
   513	$(rmcr < /home/adreyer/.cache/x11docker/x11docker-check-16958756445/share/container.log | tail)"
   514	}
   515	storeinfo containerid="$Containerid"
   516	# Wait for container to be ready
   517	for ((Count=1 ; Count<=40 ; Count++)); do
   518	  podman exec x11docker_X113_x11docker-check_16958756445 sh -c : 2>&1 | rmcr >>/home/adreyer/.cache/x11docker/x11docker-check-16958756445/share/container.log && { debugnote 'dockerrc: Container is up and running.' ; break ; } || debugnote "dockerrc: Container not ready on $Count. attempt, trying again."
   519	  rocknroll || exit 64
   520	  mysleep 0.1
   521	done
   522	
   523	# Store container output separated for stdout and stderr
   524	podman logs -f $Containerid 1>>/home/adreyer/.cache/x11docker/x11docker-check-16958756445/share/stdout 2>>/home/adreyer/.cache/x11docker/x11docker-check-16958756445/share/stderr &
   525	Dockerlogspid=$!
   526	storepid $Dockerlogspid dockerlogs
   527	
   528	# Wait for pid 1 in container
   529	for ((Count=1 ; Count<=40 ; Count++)); do
   530	  Pid1pid="$(podman inspect --format '{{.State.Pid}}' x11docker_X113_x11docker-check_16958756445 2>>/home/adreyer/.cache/x11docker/x11docker-check-16958756445/share/container.log | rmcr)"
   531	  debugnote "dockerrc: $Count. check for PID 1: $Pid1pid"
   532	  checkpid "$Pid1pid" && break
   533	  rocknroll || exit 64
   534	  mysleep 0.1
   535	done
   536	[ -z "$Pid1pid" ] && error "dockerrc(): Did not receive PID of PID1 in container.
   537	  Maybe the container immediately stopped forunknown reasons.
   538	  Output of \"docker ps | grep x11docker\":
   539	$(docker ps | grep x11docker)

It seems that podman exec x11docker_X113_x11docker-check_16958756445 sh -c : succeeds.
But after that, podman logs -f $Containerid fails.
and podman inspect --format '{{.State.Pid}}' x11docker_X113_x11docker-check_16958756445 fails, too.

I am not sure yet what is going on. Currently I suspect a podman issue, not an x11docker issue.

[mviereck]

I had x11docker --podman working and then updated today..

Did you update x11docker or podman?

Edit:
I just want to note that I don't want to fix bugs caused by podman itself.
podman is said to be an in-place replacement for docker on cli, but does not need root privileges.
If podman fails here, either needing root privileges or behaving different than docker, I don't want to fix or workaround this in x11docker.
The podman debian package is still new and in testing stage, its bugs/issues might be fixed after some time.

[mviereck]

Current state:
podman requires sysctl -w kernel.unprivileged_userns_clone=1 to work on debian.
After that, x11docker --desktop --podman x11docker/fvwm works without root privileges and it pulls the image from docker hub.
The desktop appears, so far it works.

An error is shown: id: unknown user. It seems that x11docker's user setup in container partially fails with podman although it works using docker.
--verbose gives rm: can't remove '/etc/passwd': Resource busy in containerrootrc.
This ticket indicates an issue in fuse-overlayfs that should be fixed already.

[mviereck]

Option --home does not work with podman.
Although x11docker sets uid and gid to the host user and sets --userns=host, the uid and gid are mapped to other values.
It is not possible to write to a host folder owned by the host user. Obviously podman does not regard --unserns=host.
Edit: Fixed, see below.

[mviereck]

To allow --home and --share with --podman, x11docker sets --userns=keep-id instead of default --userns=host.

• This causes podman to keep the uid and gid set by x11docker if the container user is the same as the host user.
• If the container user is set to root or other users different from host user, podman still sets up user namespacing, and the container user uid is in fact not 0.
  • To avoid user namespacing with different --user, one must also set --hostuser with same user as desired container user and run x11docker as root.
• If setting --userns=keep-id with docker then docker fails. It is a podman only option not compatible with docker.

I found this by observation, not by documentation.

[mviereck]

In mviereck/dockerfile-x11docker-mate#1 the default seup with --cap-drop ALL is reported to fail with --podman.
I cannot reproduce here. The issue might depend on system configuration, eventually sort of SELinux or apparmor restrictions.

[mviereck]

Option --gpu fails with --podman for unknown reasons. It might be related to the /etc/passwd issue.
Found in #293
Edit: --privileged allows GPU access. Maybe podman has a --device option issue?

Edit2: podman does not share /dev/dri, neither with --device nor with --privileged.
It does not print an error, it just does not do its job. A very odd bug.
GPU acceleration is impossible without the device files.

Only some nvidia setups seem to work with --privileged as reported in #293 .

Edit3: A setup with --volume /dev/dri:/dev/dri:rw --privileged allows GPU access.

[mviereck]

Current podman version 2.1.1 has a bug with options --userns and --user and is not useable yet with --podman.

[fin-ger]

I was able to get the Omnet++ IDE docker image working with the following command line:

$ x11docker --podman --cap-default --gpu -i -- --volume /dev/dri:/dev/dri:rw \
    --privileged --rm -v /home/fin/thesis/models:/root/models:Z" -- omnetpp-fin

I am on Fedora 32 with the default podman installation. I noticed that the check in lines 7321-7323 is only valid for Linux kernels which include a debian patch which was rejected by the kernel developers. This patch is not included in Fedora and many other distros which make the check fail although nothing is wrong. For my test I simply commented the mentioned lines in x11docker.

For reference, I am running an AMD graphics card with the amdgpu driver.

Dockerfile for omnetpp-fin image

Please note that this image contains a lot more than just Omnet++. I use it for a reproducible development environment of my research. In particular, OSG-Earth and Orca are additionally included. It is based on the upstream omnetpp/omnetpp-gui dockerfile.

from ubuntu:20.04 as base

run apt-get update -y && \
  DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
  git patch wget curl make ca-certificates bison flex clang perl python3 libxml2-dev \
  qt5-default libqt5opengl5-dev libgtk-3-0 libwebkit2gtk-4.0-37 default-jre gcc g++ gdb \
  mesa-utils mesa-utils-extra osgearth libosgearth-dev openscenegraph-plugin-osgearth \
  mpi-default-dev zlib1g-dev xserver-xorg-video-amdgpu && \
  apt-get clean && \
  update-alternatives --install /usr/bin/python python /usr/bin/python3.8 1 && \
  rm -rf /var/lib/apt/lists/*

# first stage - build OMNeT++ with GUI
from base as builder

arg VERSION
workdir /root
run wget https://github.com/omnetpp/omnetpp/releases/download/omnetpp-$VERSION/omnetpp-$VERSION-src-linux.tgz \
  --referer=https://omnetpp.org/ -O omnetpp-src-linux.tgz --progress=dot:giga && \
  tar xf omnetpp-src-linux.tgz && rm omnetpp-src-linux.tgz
run mv omnetpp-$VERSION omnetpp
copy patches /root/patches
workdir /root/omnetpp
env PATH /root/omnetpp/bin:$PATH
# apply patches
run find /root/patches -type f -print0 | xargs -0 patch -p1 -i
# remove unused files and build
run ./configure WITH_OSG=yes WITH_OSGEARTH=yes && \
  make -j $(nproc) MODE=debug base && make -j $(nproc) MODE=release base && \
  rm -r doc out test samples config.log config.status

# second stage - copy only the final binaries (to get rid of the 'out' folder and reduce the image size)
from base

arg VERSION
env OPP_VER=$VERSION
run apt-get update && apt-get install -y python3-pip xvfb
run pip3 install plotly pandas psutil requests
run wget https://github.com/plotly/orca/releases/download/v1.3.1/orca-1.3.1.AppImage && chmod +x /orca-1.3.1.AppImage
run echo "#!/bin/sh\nxvfb-run -a /orca-1.3.1.AppImage --appimage-extract-and-run --no-sandbox \"\$@\"\n" > /usr/bin/orca && chmod +x /usr/bin/orca
run mkdir -p /root/omnetpp
workdir /root/omnetpp
copy --from=builder /root/omnetpp/ .
env PATH /root/omnetpp/bin:$PATH
run chmod 775 /root/ && \
    mkdir -p /root/models && \
    chmod 775 /root/models && \
    touch ide/error.log && chmod 666 ide/error.log && \
    rm -f bin/omnest && \
    mv bin/omnetpp bin/omnetpp.bak && \
    sed 's!-Dosgi.instance.area.default=$IDEDIR/../samples!-Duser.home=/root/models -Dosgi.instance.area=@user.home!' bin/omnetpp.bak >bin/omnetpp && \
    rm bin/omnetpp.bak && chmod +x bin/omnetpp
workdir /root/models
run echo 'PS1="omnetpp-gui-$OPP_VER:\w\$ "' >> /root/.bashrc && chmod +x /root/.bashrc && \
    touch /root/.hushlogin
env HOME=/root/
env XDG_RUNTIME_DIR=/tmp/xdg-runtime-root
cmd /bin/bash --init-file /root/.bashrc

[mviereck]

@fin-ger Thank you for the detailed report!

I have inverted the check so it will not fail if the file does not exist at all.
From:[ "$(cat /proc/sys/kernel/unprivileged_userns_clone)" = "1" ] || error
To: [ "$(cat /proc/sys/kernel/unprivileged_userns_clone)" = "0" ] && error

[mviereck]

I've deprecated option --podman in favor of --backend=podman to allow a more general approach. This should allow other commands like nerdctl (containerd), too.

[mviereck]

Good news: Currently I encounter no issues with --backend=podman.
Even options --gpu, --home and --init=systemd work as they should (had issues before).
Currently I am not aware of drawbacks compared to docker.

podman version 3.0.1 on Debian testing.

[mviereck]

With new release v6.9.0 x11docker supports docker, podman and nerdctl, all of them in rootless and rootful mode.
Most of these setups are barely tested yet, please give feedback if something wents wrong.

So far, I consider this task as solved. :-)