waiting very long for "x11docker=ready" | Xorg in rootless podman | bubblewrap setups #466
Comments
|
Also tried x11docker/openbox and x11docker/kde-plasma, and I get the same looping behavior. |
|
The above turned out to be a python issue (I was trying to use a containerized python, and it wasn't working right). After I installed python3 (uncontainerized), I get a different problem: |
|
Thank you for the report!
Was this a custom setup of yours or something more common that x11docker should check and be able to handle?
Likely x11docker wants to run |
|
The setup is very pure: Start with Fedora CoreOS (which comes with podman), install absolutely nothing else on it (although I needed python for your script - more on that later...), and use the x11docker script with xserver container and one or more window manager or desktop environment containers to get a choice of desktop enviornments running on it. I can use either VirtualBox or Qemu/KVM to house the CoreOS install for experimentation, but eventually my goal is for a bare metal install of CoreOS (with no additional installs on it) with a fully podman containerized single user no remote access desktop environment. I will try BTW: about x11docker script requirement for python. It seems the requirement is very light. Possibly the script would work with just using |
Oh, right, you are already using TTY switching works fine without root.
You can reduce the Dockerfile of
I am not entirely happy about the |
|
Running |
Ok, right; I confused a bit here, sorry. Now I remember: Running Xorg within a container works only with rootful podman. To avoid the need of |
mviereck
changed the title
|
Isn't the Note that a wayland-based x11docker desktop such as x11docker/kde-plasma does no better. I have seen that Fedora Silverblue and Kinoite (the KDE variety of the same family) run wayland as the user, not as root. So that suggests the possibility of a container doing so on CoreOS. But, using x11docker/kde-plasma as the target desktop container did not change matters - I again get the same |
|
BTW: if you want to get CoreOS up and running in VirtualBox quickly, I can help with that. It's not your typical install-from-ISO-as-CD distro. |
You don't need to specify option
I am not sure yet if I understand right. Or maybe I misunderstand you? Please show me your commands and the resulting error messages.
It doesn't matter which image you use because x11docker sets up Wayland or X before the container is started. |
Thank you for offer! I'll try to reproduce in my regular fedora VM first. |
|
Running Running in the x11docker git clone directory: This produces error message. I can't copy-n-paste it or transfer it out of the VirtualBox VM easily with CoreOS, because no VirtualBox guest extensions are present. If you need the whole output and/or log file, I will work on a transfer ability. If I instead run that's when I get in the ~/.cache/x11docker/x11docker.log file If I need other privileges to run rootless, I can worry about getting them later. |
Thank you! I don't need the full text, I just needed to sort the commands and their error messages. The first error is correct:
Please provide Your second command |
|
I did a |
|
I'll bet it isn't finding x11docker/fluxbox or any other desktop the user has pulled. Maybe I should run two x11docker instances, one as root for the server the other as user for the desktop? Is there a way to do that? |
|
Surprisingly, the fluxbox desktop appeared after a very long delay. I didn't need to start a separate x11docker instance as I thought, just waiting. Hopefully this long delay is a one-time issue. |
|
I've tried to reproduce the issue but now have the unrelated problem that my root partition has not enough space left for image x11docker/xserver to pull it in rootfull podman. If you somehow can provide me the log file ~/.cache/x11docker/x11docker.log , I might find a hint what is blocking the process.
At least CTRL+ALT+F(n) should be possible to switch to another console.
That is basically possible, but should not be needed.
You could specify another tty with option It is late here, and I am tired. I'll look at this tomorrow again. |
|
I have both fluxbox and openbox working, but not kde-plasma. It looks like the x11docker/fluxbox and x11docker/openbox containers have no X client apps, such as xterm, in them. Of course they are designed to be used as base layers for other containers, so I will do that. I noticed that on distros where I can start X as non-root (via the startx -> xinit route), they have Xorg.wrap, a setuid for doing just that. But a more secure scheme with just the necessary capabilities is probably possible: some setcap'ed way of launching the x11docker script as the user instead of root. |
Do you still have the long startup delay? It can be as well a podman issue that I once had, too. It can be solved temporary with
kde-plasma needs Also add option
The ideal way would be to be able to run Xorg with rootless podman. We might give it a try again and ask the podman developers for help. x11docker already gives the bare minimum of capabilities to the container of |
|
I tried I will keep I don't think Xorg.wrap would fail to run just because it is in a rootless container. But it would fail to deliver the necessary capabilities, which the rootless container didn't inherit from the user. I think that the necessary capabilities have to be delivered from the outside-in starting with the x11docker script itself. However, I know only enough about capabilities to know I don't know enough about capabilities :( But I do think this can be fixed externally to podman, assuming podman doesn't intentionally drop excess capabilities it has inherited when run rootless. |
|
And mate is finally running! After over an hour of that exe process doing whatever. |
|
Do you have a way to send me
x11docker doesn't run anything called |
|
Once they finally start up the first time, each desktop has pretty fast (~10sec) startup times after that. So I will have to try a new one. I will try fvwm and send you the log if it is slow. If that isn't slow, I will try XFCE or LXDE. I figured out a way to get files out of a vbox without guest additions by using a USB drive. I would rather have a way to mount something rw in the vbox that is ro outside, and there's probably a way to do that, but the USB drive trick works without further thought, so it wins. I think that |
You mean, once you have waited a long time for the first startup, later startups with x11docker are fast? |
|
fvwm was fast as user. Nothing interesting in the log, but I've saved it in case it may prove useful by comparison to something slow.
Yes. On to XFCE as user. If that's slow, I'll try it as root. Actually, I will pull it as root, then Also, if it is slow, I will try |
|
Just an idea: Maybe podman is pulling the image again instead of using the one you have pulled before? |
There's no network activity, either. I just had to forcibly stop the vm that was trying to start XFCE - I was logged in on another console as root spying on it with pstree and other things, and something I did caused the vm to go crazy. But, I did see the I'll take a look at the log file... |
|
No log file. So I restarted XFCE, and will be a bit more careful while spying on it. doing |
Sorry, I should have checked it better before. I have added a new inofficial option Yet you should be able to run rootless |
|
Still similar problems. Log file attached... |
|
This time you get an Xorg error, same as me now: At least x11docker does not forbid running Xorg in a rootless container. That was the intention of my changes above. On this base further experimental code might be added for tests with Xorg in rootless podman. btw, to see all and only the the Xorg messages, you can use inofficial option |
|
On Alpine, where --unshare-all (which includes --unshare-pid) works when rootless bwraping Xorg, the Xorg version is newer than on Debian where it doesn't work to --unshare-pid. Also, Debian had loaded one extra extension: SELinux. But using Xorg's -extension arg to remove that had no impact on the inability to --unshare-pid. The Xorg versions are: I have a feeling that the Alpine developers changed something (hence that last extra '.4') and compiled Xorg themselves, considering the other ways Alpine differs from almost every other distro by relying on busybox and musl. What that might mean is that if you want rootless capability in more places, it might be worth the effort to create an Alpine version of x11docker/xserver. That's just a theory. At the very least it might enable --unshare-pid. |
|
As a first quick attempt I've build an I'll give an alpine based image a try. It might have the nice side effect of creating a smaller image overall. |
|
Did you try it without unsharing pid? |
No; using less namespacing would only ease things, not break them. Yet I run some tests with an alpine based image, but I am running into the same error message Edit: |
True. But do we know yet whether the problem is due to the pid namespace, groups, or even if there is something in /dev that should have been mounted? I know the error messages say certain things, but I wonder if Xorg tries several different things and would take the first success, if none succeed it only reports the final failure. Meaning that a possible successful route might not involve fixing what that error messages implies is the culprit. |
Basically we can compare with a successful startup of Xorg in rootful podman. The Xorg log in rootless podman should be identical once all issues are solved.
The list of shared devices is valid for rootful podman, so it must be valid for rootless podman as well. Currently I assume your attempt with bubblewrap is the way to go. It might be just impossible to run Xorg in rootless podman. |
Maybe so. I'd still like to unshare pid everywhere. Maybe bubblewrap with an Alpine chroot on Debian is worth a try to see where the problem is. Also, I like your idea of bundling several window servers together: Xorg, Xephyr, Wayland, XWayland, Xpra, etc. all in the same chroot. Maybe as a flatpak, which uses bubblewrap. |
|
On Alpine, even though I can bwrap Xorg with |
That's really weird. On the first glance I cannot see how this could happen, and why using namespacing would make a difference here.
In general that should be possible.
In the |
The buffering at the tty is the normal buffering when keypresses occur during a previous command execution. In this case, that previous command is my startx-like script launching Xorg and the window manager. If I background that with |
|
I have lately been trying to sandbox weston-launch, and found that, like Xorg, it does not like unsharing the pid namespace. In addition, it does not like unsharing the cgroups namespace, either. This is on Debian. However, non-top-level weston (backed by either Xorg or wayland) and Xwayland sandbox very well with all namespaces unshared. They also do not lose much performance when sandboxed. A fully sandboxed weston/Xwayland combination is faster than an unsandboxed Xephyr or Xpra, at least in my benchmarks. I am using |
|
Sorry that I did not respond for a long time! |
Ok, but should I keep posting to this issue? My goal is a portable sandboxed desktop, including sandboxed display server, that is agnostic about window managers, and that can run on minimal distros that are more secure than typical (because immutable like CoreOS, small attack surface like Alpine, etc.). The target audience is single-login desktop metal users, not Docker users. Although there is overlap, I would guess that most of your audience cares more about the ability to run GUI apps from containers without lowering the security on those containers, and not so much about whether the display server (X or Wayland) is itself in a sandbox. So, there is no need to apologize about not responding timely! You have already paid more attention to this issue than I anticipated, and I am grateful. |
Yes, please!
You are right. Running X or Wayland in container adds some isolation, but still needs so much privileges that only small improvement of security is possible.
I am not sure what you mean with "agnostic of window managers". |
No part of the sandbox setup or underlying distro depends on the user's choice of window manager and server pair. The user should be able to choose dwm on Xorg or KDE on Wayland, or pretty much anything in between, and switch easily without installing anything in the underlying distro (especially if it is immutable). This should be an easy thing to accomplish: package WM or DE as flatpak or docker/podman container, as you do. I add it as a goal to rule out any choices that are specific to some window managers or desktop environments but not others, even though I don't anticipate any. I like what Fedora did with Silverblue and Kinoite, but I think they missed an opportunity by not moving the whole DE and display server into one or more flatpaks or podman containers. The user has to choose the DE they want at installation, cannot easily switch, and the OS and DE upgrade together - so these are a counterexample to the agnostic goal above. But, Fedora also has CoreOS without any WM/DE - so that was my initial target, CoreOS with WM/DEs and servers in flatpaks or podman containers. |
mviereck
changed the title
I was experimenting to see if it's possible to run wayland contained with bwrap in a similar way Initially trying this in a VT with non-root user: In this bash environment I can see the home folder is the host contents of ~/bwhome, and real home is hidden, and I can confirm there is no network capabilities (see Narrowed it to it escaping if it has a |
|
@digitalsignalperson |
|
@jonleivent thanks for the links, I hadn't heard of labwc. It's surprising how dbus gives access to everything, and the list of wayland/wlroots protocols that can allow for monitoring/messing with many things (as described in the labwc discussion & with wl_monitor.py). Do you have examples/snippets of your bwrap scripts you could share? E.g. how to run a sandboxed labwc, or how to run another client with dbus-run-session in a sandbox. For isolating Xorg in general, some ideas here might be interesting/relevant:
|
For labwc, the only devs that need to be bound into the bwrap sandbox are It's certainly possible to block out much more from the sandbox, but I haven't investigated that yet. I do have a labwc AppArmor profile that probably makes blocking out other things from the sandbox unnecessary. I also have a seccomp bpf for it , which is slightly more permissive than the seccomp bpf I use for everything else. I haven't tuned those tightly yet either. Of course, if you want to be able to launch apps from labwc's menu, you'll need to allow that somehow. I do that with a background process running external to the sandbox that listens on a named pipe for requests to launch apps, and then only launches apps that are in a specific write-protected folder just for it. Which are all scripts that filter arguments passed in and start other sandboxes. The named pipe is then The Interesting links. I once tried a multi-VM configuration, and found it was a huge consumer of resources. I don't have the hardware necessary to use that effectively. Also, probably my security requirements are likely lower than a typical Qubes-phile. |
|
@jonleivent thanks for all those details. I notice with that bwrap command binding $XDG_RUNTIME_DIR, it's possible to escape the sandbox through dbus, even if labwc doesn't use dbus. I ran that command with If I don't bind XDG_RUNTIME_DIR, it can't start labwc (or any other wayland compositor I've tried). labwc in particular: If I do bind XDG_RUNTIME_DIR, monitoring Ideas for approaches to get a seat without binding XDG_RUNTIME_DIR or without unrestricted dbus access:
For X11, I don't have any luck with the bwrap commands. I can use the same command as above for labwc, but use startx: First I get "Cannot open virtual console 1 (Permission denied)". If I open that up with chmod, I get "Switching VT failed". If I try startx with a different VT like
I'm curious about this. Do I read this as you aren't running apps that are binded into the sandbox in /bin, but purposefully running apps outside the sandbox? |
That doesn't happen in my case because I don't run a session dbus. In other words, I have patched the login startup for that user so it doesn't auto-launch the session dbus. I did this by masking the systemd unit for that user, so that no systemd services run for that user. There is nothing in that user's $XDG_RUNTIME_DIR initially. You can always bind something other than $XDG_RUNTIME_DIR in to the sandbox as $XDG_RUNTIME_DIR, like:
All of my apps, with just one exception (a terminal), run in their own private sandboxes. Obviously, you can't start a bwrap sandbox directly from within an existing bwrap sandbox. Instead, you need to have something outside any sandbox that can start new sandboxes - a background process (daemon). When I raise labwc's primary menu, I have entries on it to for instance run firefox. I want that to run in its own sandbox which has entirely different permissions than the one that labwc runs in. In fact, labwc's sandbox has no network connection, so firefox wouldn't run there. What I have labwc's menu do is instead of directly exec'ing firefox, it writes a command to run firefox out on a named pipe that the daemon is monitoring. The daemon reads that, and launches a script that runs firefox in its own sandbox. The scripts that the daemon is allowed to run (17 so far) are very restricted, so that no sandbox can use them to escape in a bad way (such as run arbitrary code outside any sandbox). I may convert this later to using a unix socket instead of a named pipe (if for instance I need bidirectional communication, or the ability to pass fds), but for now the named pipe works well and is very easy to set up. |
|
@jonleivent thanks for explaining & sharing some details of your unique setup. It would be interesting to play with it in a VM and poke around a working demo or sounds like good material for a blog post too.
Is nested bwrap not possible (PR landed here containers/bubblewrap#164) or is that a design decision for your setup? Still curious about getting the seat backend part for wayland to work, and also getting sandboxed x11 to work. I'm on an arch linux setup. Reading above about this working on alpine, I managed to get a working POC in a VM: lxc launch images:alpine/3.18 alp --vm -c limits.cpu=4 -c limits.memory=4GiB --console=vga -c security.secureboot=false
# from another terminal set root password
lxc exec alp -- /bin/ash
passwd
# log in as root in the vga console
setup-xorg-base
apk add openbox xterm font-terminus
apk add bubblewrap
adduser person
addgroup person input
addgroup person video
exit
# login as person
# run X without sandbox - works
startx /usr/bin/openbox-session
# with sandbox - also works
bwrap --ro-bind / / --dev /dev --proc /proc --dev-bind /dev/dri /dev/dir --dev-bind /dev/input /dev/input \
--tmpfs /tmp --tmpfs /var/tmp --tmpfs /var/cache --ro-bind /var/cache/fontconfig /var/cache/fontconfig \
--tmpfs $HOME --unshare-all --new-session -- startx /usr/bin/openbox-sessionDid you also get it to work with debian? I cannot lxc launch images:debian/buster bust --vm -c limits.cpu=4 -c limits.memory=4GiB --console=vga -c security.secureboot=false
# from another terminal set root password
lxc exec bust -- /bin/bash
passwd
# log in as root in the vga console
apt install openbox xinit
apt install xterm
apt install bubblewrap
adduser person
addgroup person input
addgroup person video
addgroup person sudo
exit
# login as person
# run X without sandbox - works
startx /usr/bin/openbox-session
bwrap --ro-bind / / --dev /dev --proc /proc --dev-bind /dev/dri /dev/dir --dev-bind /dev/input /dev/input \
--tmpfs /tmp --tmpfs /var/tmp --tmpfs /var/cache --ro-bind /var/cache/fontconfig /var/cache/fontconfig \
--tmpfs $HOME --unshare-all --new-session -- startx /usr/bin/openbox-session
# cannot open /dev/tty0
# debian is weird, startx seems to be non-blocking
sudo chown person:tty /dev/tty2
bwrap --ro-bind / / --dev /dev --proc /proc --dev-bind /dev/tty2 /dev/tty2 --dev-bind /dev/dri /dev/dir --dev-bind /dev/input /dev/input \
--tmpfs /tmp --tmpfs /var/tmp --tmpfs /var/cache --ro-bind /var/cache/fontconfig /var/cache/fontconfig \
--tmpfs $HOME --unshare-all --new-session -- startx /usr/bin/openbox-session -- vt2
# xf86EnableIOPorts: failed to set IOPL for I/O (Operation not permitted) |
Even if it is possible, the inner bwrap can't get more capabilities than the outer one. As I mentioned, my labwc sandbox has no network connection. If I started firefox in a nested bwrap inside that, it wouldn't have a network connection either. Also I disable unprivileged (actually, all) user namespaces, which makes it impossible to nest because bwrap always sets no-new-privileges, and running bwrap without unprivileged user namespaces needs setuid, which is disabled under no-new-privileges. And, something to remember about sandboxes. They prevent things from getting out, not in. With nested sandboxes, the inner apps are not protected from the more outer ones. The outer ones can do things like use the magic links in /proc/PID/fd of an inner app. Or ptrace it (actually, I have Yama turned up so ptrace doesn't work except for root - but if I didn't...!). I abandoned my alpine setup a while ago. It was flaky in strange ways. The tty console always received input even if that input was intended for X11 when I had X11 bwrapped. I would start X11 in a bwrap and have fluxbox running, then start an xterm in that and type into the xterm. When I exited X11, I would see that input I typed in the tty console. Alpine is just not like other nixes. I'm on Debian 12, and will probably stay there for a while. I have also been entirely focused on wayland because my experiments with using nested X11 servers (like Xephyr and Xpra) to isolate keyboard/mouse/clipboard/etc from apps was disappointing. It worked, but was difficult to manage and lost graphics performance. |
In Fedora CoreOS (in a virtualbox VM), with latest images of docker.io/x11docker/xserver and docker.io/x11docker/fluxbox, using the 7.4.2 version of x11docker script:
x11docker -D -V --backend=podman --desktop x11docker/fluxboxloops apparently forever with:
I'm assuming docker.io/x11docker/{xserver,fluxbox} are your images. Am I wrong? Also, doesn't the x11docker script test such things?
The text was updated successfully, but these errors were encountered: