Running “full” OS with Lithos

[jirutka]

Hi,

it seems that Lithos is designed for running services that may be isolated using namespaces, cgroups, capabilities(?), i.e. running them inside “containers.” I wonder, can I use it even to run ”full OS” (Alpine Linux to be specific) with traditional init system (OpenRC) and multiple services, including OpenSSH server for users to connect into? Like with LXC that I currently use. Theoretically it should be possible, but it seems that it’s really not designed for such use case (?), so are there any limitations or design decisions that makes it really bad idea?

[tailhook]

Hi!

Technically you can. There are a few quirks, though:

1. We don't have network isolation yet. We have a short-term plan for basic support of handing network namespace to the container (and setting up it with some scripts outside of the scope of lithos). Also, we're looking towards supporting CNI plugins, but I'm not sure I'll have time to try it anytime soon.

2. Lithos runs its own process (lithos_knot) as pid1 in container. This helps applications, but it may be a problem for systemd, or any other init. I'm not sure how easy to fix this one (because we use lithos_knot process' command-line to find out which processes are running on restart, this mechanism needs to be changed to run a full-featured init).

3. It's probably super-tedious to setup. In docker or lxc you have everything writable by default. In lithos, you have nothing writable by default. And you need to add every tiny little directory that needs to be writable to lithos config (and presumably put it outside of the container image). Even if it's just to track ntpd state. Many utilities run as root are trying to write to /etc. There are probably much more quirks for systemd (like setting container_uuid environment variable). And maybe a little bit less for openrc. Adding overlayfs making filesystem kinda writable isn't too hard, though.

4. We also don't have capabilities support (we want to have, though). So the recommended way is not to run as root in container. Or to use user namespaces, which have it's own root with somewhat non-superuser capabilities. Latter may be problem for systemd/openrc too.

All in all, we can add some features incrementally. But I don't see a reason to make full OS a first-class citizen for lithos. There's neither space where we can improve a lot (any specific issues with LXC/LXD?) nor there are lots of use cases for such containerization in the wild I think.

(on the other if someone reading this wants to use underlying libmount and unshare crates for running full OS images I'm very enthusiastic to this idea, so let me know if you need any help)

[jirutka]

Thanks for very thorough and quick answer!

Well, LXC works, but I don’t like that project. LXC is 42k LoC in C, that’s really massive for something that can be (basically) written even in dozens lines of shell with few tiny, single-purpose utilities like unshare, ip etc. Also it has very cryptic error messages, so when something get wrong, it’s quite hard to find the cause.

I’d like to find same sane, lightweight alternative, preferably written in some decent language made for system programming (I mean not in Go…). Vagga and Lithos looks very promising.

I have both use cases. I need to run isolated services, but also some full containers that I can give to users, so they can manage some their services inside.

[tailhook]

I need to run isolated services, but also some full containers that I can give to users, so they can manage some their services inside.

Well, technically we have a concept of sandboxes just for that. I.e. you allow users to upload images to certain directory configured in a sandbox. And arrange some script to update their /etc/lithos/processes/<sandbox-name>.yaml. This allows them to freely update their images, add and remove services, but they can't escape a sandbox (basically a set of directories they are allowed to see, and a set of user ids they are allowed to use, etc.).

To be perfectly clear: it doesn't help to run full OS. But it does help to manage services without running a full OS. It's easy to run sshd as a service in the lithos, if you want users to be able to log in for some simple tasks like looking system stats or grepping logs.

For generating processes yaml without allowing users to access host system you may use consul-template or confd, or some wrapper around whatever provisioning system you have (e.g. ansible)
(we're developing a verwalter for this purpose, but it still lacks good auth at the moment)

[tailhook]

@jirutka, any questions left here?

[jirutka]

No questions left, thank you for the answer! We can close this now.