blog/2021-03-25_proxmox_docker/

[utterances-bot]

Running docker inside an unprivileged LXC container on Proxmox - du.nkel.dev

A personal code notes blog

https://du.nkel.dev/blog/2021-03-25_proxmox_docker/

[Sieboldianus]

Comments welcome. If you do not want to allow Github API access for giscus.app, feel free to directly add comments on Github.

[jm009]

Thank you very much for your report, that running docker inside an unprivileged linux container (lxc) worked for you.
Unfortunately it does not work when I tried it. I get

root@TEST-LXC:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete 
Digest: sha256:97a379f4f88575512824f3b352bc03cd75e239179eea0fecc38e597b2209f49a
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown.
ERRO[0004] error waiting for container: context canceled 
root@TEST-LXC:~#

For more details please see https://forums.docker.com/t/docker-problem-in-unpriviledged-lxc-on-debian-11-2-bullseye

[jm009]

Hello Alexander Dunkel/Sieboldianus,
thank you again for your report. After not being able to reproduce it on the first try (I do not have Proxmox, just a Debian command line), and after reading carefully your report a second time, I would like to ask:
Are you sure, this is a Linux container (isolated by Linux kerne features), and not rather a virtual machine (isolated by CPU features)? I am asking this, because I wonder
a) if the configuration
rootfs: storagedocker:100/vm-100-disk-0.raw,size=60G
does create a virtual disk, instead of using a subdirectory of the host (as I know it from Linux containers on my system)
b) I wonder, why in the configuration I do not see userid/groupid mappings, that are used for Linux containers ("user namespaces")
Could you check this again? For a Linux container I usually do lxc console or lxc attach to log into the virtual host, and not ssh, like you mention (however ssh should be available, if sshd is running in the Linux container).
Thank you again, and best regards
Jan Michael Greiner

[Sieboldianus]

Sorry, you are obviously using a different system. The post above is for Unprivileged LXCs on Proxmox. While native LXCs are somewhat similar, Proxmox introduces special configuration files for isolating LXCs (e.g. uid/gid mapping). I don't use native LXCs on Debian.

[jmbenevise]

Well done.
I works well
I was extremely happy...
Until I try to clone the container (LXC)

[jmbenevise]

I may stay on PVE 6.4 a bit longer :-(

[Sieboldianus]

This has nothing to do with 6.4/7.10 - only if you're using ZFS - and if so, cloning would not have worked on 6.4 either, if you're using a docker volume mount.

[jmbenevise]

Agreed.
But with 6.4, I can use a regular mount point with AUFS.
Which is how containers in LXC are built in my 6.4 cluster.

[Sieboldianus]

I see what you mean. Since I have migrated to ZFS at the same time while moving from 6.4 to 7.1, I did not see that the AUFS Kernel module seems deprecated in 7.1, as is suggested here.

I can say that I successfully used/installed/tested Docker in 7.1-10 without the volume mount (i.e. following the first part of this guide), backed by Directory Storage, so it appears AUFS is perhaps not needed.

[jmbenevise]

I will wait until eof of Proxmox 6.4, and will eventually change my config if AUFS or fuse-overlayfs are not available.

[rnz]

not working with docker swarm

[rnz]

# pveversion 
pve-manager/7.1-12/b3c09de3 (running kernel: 5.13.19-6-pve)

[rnz]

@Sieboldianus you can verify this by doing (on proxmox 7.1) install portainer (for example) in docker swarm:
https://docs.portainer.io/v/ce-2.9/start/install/server/swarm/linux

[Sieboldianus]

I don't use portainer, but I can verify Docker Swarm on Bullseye 11 LXC (unprivileged) on the weekend perhaps. I have various Docker Swarm CI's at work, but none at home yet. Shouldn't be much more than docker swarm init.

[Edit] I see, you want me to test deploying Portainer in swarm. 👍

[rnz]

For work of services need enable net.ipv4.ip_forward=1 on host

[Sieboldianus]

I tried Portainer once, it looks nice, but I don't see a real benefit over docker-compose (or Deployment in Swarm through a Gitlab Runner), I won't work on this further. Also, modifying the hypervisor would be a no-go, since this could affect other stable services.

[frezeen]

what about migration to nodes? it dont work for me

[Sieboldianus]

Good point. I will add this to the list of caveats. Migration will need manual work because the docker zfs-xfs volume needs to be moved, too.
But what is the application for this? I would strongly argue against using Unprivileged LXC + Docker in a distributed setup. I would expect that those who run distributed setups with Proxmox can afford the hardware to use a dedicated VM for Docker - and this will be also easy to migrate.

[frezeen]

Lxc has vantage of CPU core assigment on cluster...u are not limited to set core numbers of node that have less... VM use all assigment Memory ,and when u have a 64gb VM its slow to replicate or move 64gb cached RAM..example, My 64gb docker VM full all Memory for cache in proxmozx in less than 24 hours...My lxc with same 53 containers...use less less Memory. Replication and HA Is more fast. Anyways this Is hard to do with a priviliged docker lxc too..overlay+lxc+docker+replication+ha its impossible for me, until a solution

[Sieboldianus]

Lots of good reasons to use LXC over VM. If you find a solution to do the automatic migrations with LXC+Docker, please update here.

[EcurbNosneb]

Went the fuse-overlayfs route:
proxmox lxc unprivileged, fuse=1,keyctl=1,nesting=1, ubuntu 20.04
apt install fuse-overlayfs
apt install docker.io

Have portainer and home_assistant running.

Replication works, proxmox backup-server backups work.

Migration had a few hiccups. First try got the message "can't get lock". Stopped container, migrated w/o incident, restarted container and it was fine. Then migrated back to the first node and it was fine (stopped and started itself fine). Subsequent migrations would periodically look successful but docker would lock up (docker info or docker ps would not return for example). Seems rock solid when not migrated, but when migrated needs to be checked afterwards.

On my home lab (3 nodes: A10-5800K, i5-2500, FX-8350 totaling 16 cpus, 56gb ram, 12tb of SSD, 9 total LXCs) VM+docker+containers was 4gb ram, 35GB disk where LXC+docker+containers is 512K ram, 3.3gb disk. On my home lab VMs are resource hogs (I no longer have any windows servers - yea!). Lxc is just a breadth of fresh air and simplicity using proxmox.

Thanks for sharing.

[EcurbNosneb]

Forgot to mention: everything running on ZFS mirrored SSDs.

[Sieboldianus]

This is great, thanks for sharing! I am unsure where the reports of deadlocks with fuse-fs+proxmox/lxc+docker come from, maybe it is just a thing of the past. Portainer and Home Assistant are more of the lightweight container sort, so would be interesting to see if db/write-heavy systems work equally well (Gitlab, Nextcloud). Nonetheless, this is a good signal. 👍

[Sieboldianus]

Linked to your comment from the specific section in the post above.

[EcurbNosneb]

I ended up turning off replication and backups as when they were enabled my proxmox node would periodically (about once a week) lock up. Timestamps always showed the start of a backup or a replication. Once off, no more lockups but no replication nor backups.

[chribro88]

So I'm coming from the LXD side, but I've found your blog very useful.

One thing I am getting stuck on, and seems that has also caught you is the host messages about "overlayfs: upper fs is missing required features" - also the post here
https://forum.proxmox.com/threads/docker-lxc-unprivileged-container-on-proxmox-7-with-zfs.99796/post-451866

For me I've had this message trying different set ups (zvol formatted as xfs/ext4 and mounted in LXC and also separate Directory pool on EXT4).

I have noticed thou if I make the LXC privileged, there's no message and "Native Overlay Diff" is set to true

So it would seem there some kind of permission settings that need to changed to get this working 100% without that message?

[Sieboldianus]

My (limited) insight on this one: OverlayFS uses "layers" to merge multiple dirctories (e.g. Docker image Layers) to present them as one, see How the overlay2 driver works

Now, when ZFS is behind the scene, these features won't work in unprivileged mode because root-access to the host is required (because some restricted Kernel features seem to be used, see a good description here). If you run in privileged mode, you grant docker root access to your host, and you give up any security benefits from the LXC nesting, which may or may not be fine (for me, it isn't). Warnings such as missing RENAME_WHITEOUT (etc.) are there for the same reason.

On the other hand, think hard whether these features are really needed: They may be crucual in a multi-tenant, enterprise, or high-workload setting (like 100-1000 Docker runs a day) because Docker can deduplicate its layers. My observation was that the performance-impact is negligible in a home server or private setting. There is no feature impact, that means everything will work, just that pulling and updating docker images may be a bit slower (which happens only once a day for me, e.g. at night, when Watchtower may pull new updates; or when I am triggering CI to update my websites).

I suggest, if you cannot run in unprivileged mode, always choose a VM (over privileged mode), to nest Docker inside, no big deal.

[hereisderek]

I'm still not able to enable overlayfs2 after all the step, running on:
pve-manager/7.2-11/b76d3178 (running kernel: 5.15.64-1-pve)

on host:

root@prox:/# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

wireguard
overlay
#aufs
#fuse
zram num_devices=3
root@prox:/# lsmod|grep -E "overlay|aufs"
overlay               147456  0
root@prox:/#

on lxc:

root@media01:~# lsmod|grep -E "overlay|aufs"
overlay               147456  0
root@media01:~# docker -v
Docker version 20.10.21, build baeda1f
root@media01:~# cat /etc/docker/daemon.json
{
  "storage-driver": "overlay2"
}
root@media01:~# systemctl start docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xeu docker.service" for details.

Oct 27 12:17:04 media01 systemd[1]: docker.service: Scheduled restart job, restart counter is at 1.
Oct 27 12:17:04 media01 systemd[1]: Stopped Docker Application Container Engine.
Oct 27 12:17:04 media01 systemd[1]: Starting Docker Application Container Engine...
Oct 27 12:17:04 media01 dockerd[1795]: time="2022-10-27T12:17:04.666389468+13:00" level=info msg="Starting up"
Oct 27 12:17:04 media01 dockerd[1795]: time="2022-10-27T12:17:04.669628730+13:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Oct 27 12:17:04 media01 dockerd[1795]: time="2022-10-27T12:17:04.669689697+13:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Oct 27 12:17:04 media01 dockerd[1795]: time="2022-10-27T12:17:04.669749536+13:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Oct 27 12:17:04 media01 dockerd[1795]: time="2022-10-27T12:17:04.669784793+13:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Oct 27 12:17:04 media01 dockerd[1795]: time="2022-10-27T12:17:04.672384894+13:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Oct 27 12:17:04 media01 dockerd[1795]: time="2022-10-27T12:17:04.672454165+13:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Oct 27 12:17:04 media01 dockerd[1795]: time="2022-10-27T12:17:04.672516785+13:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Oct 27 12:17:04 media01 dockerd[1795]: time="2022-10-27T12:17:04.672551399+13:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Oct 27 12:17:04 media01 dockerd[1795]: time="2022-10-27T12:17:04.677366078+13:00" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2
Oct 27 12:17:04 media01 dockerd[1795]: failed to start daemon: error initializing graphdriver: driver not supported
Oct 27 12:17:04 media01 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE

[hereisderek]

here is the container config i used

arch: amd64
cores: 4
features: fuse=1,keyctl=1,mknod=1,nesting=1
hostname: media01
memory: 4096
mp0: /mnt/zram/lcx/media01,mp=/mnt/cache,replicate=0 					#cache
mp1: /mnt/zram/lcx/media01/var,mp=/var/cache,replicate=0 				#cache
mp2: /storage_3t/data/tmp/lxc/media01,mp=/mnt/cache/disk,replicate=0	#cache
mp3: /root/workspace/data/lxc/media01/srv,mp=/srv,replicate=0
mp4: /storage_3t/media,mp=/mnt/local/Media,replicate=0
mp5: /storage_3t/data/saltbox/data,mp=/mnt/data
net0: name=eth0,bridge=vmbr0,gw=192.168.50.1,hwaddr=7A:3B:77:48:BA:E6,ip=192.168.50.7/24,ip6=auto,type=veth
ostype: ubuntu
rootfs: local-zfs:subvol-102-disk-0,size=2G
swap: 1024
unprivileged: 1
lxc.idmap: u 0 100000 1000
lxc.idmap: g 0 100000 1000
lxc.idmap: u 1000 114 1
lxc.idmap: g 1000 122 1
lxc.idmap: u 1001 101001 64535
lxc.idmap: g 1001 101001 64535

i also removed those id mapping and mount points, still to no avail

arch: amd64
cores: 4
features: fuse=1,keyctl=1,nesting=1
hostname: media01
memory: 4096
net0: name=eth0,bridge=vmbr0,gw=192.168.50.1,hwaddr=7A:3B:77:48:BA:E6,ip=192.168.50.7/24,ip6=auto,type=veth
ostype: ubuntu
rootfs: local-zfs:subvol-102-disk-0,size=2G
swap: 1024
unprivileged: 1

[Sieboldianus]

Looks like you're using ZFS. I don't see a mountpoint for mp=/var/lib/docker - which approach did you use to prepare the storage for Docker+ZFS?

[hereisderek]

it's just a plain boring container running on a proxmox that's running on a zfs as root, i did a few trails with not mounting specially for /var/lib/docker at all, so it will all be part of a data vol created by the host, and manually create a folder mount in the container at /var/lib/docker, and create a new volume just for that, however none had worked.

[hereisderek]

i guess it's the first approach, couldn't get the overlay2 working

[Sieboldianus]

What is the dmesg output from the host?

[iGadget]

After many days of tinkering (and failing), here's what I found.

zfs create -s -V 8G tank_ssd/lxc/docker

If you maintain the Proxmox naming scheme for zvols, i.e. vm-101-disk-1, instead of naming the zvol docker, it will show up in the PVE WebGUI and attach itself to the LXC's config with the corresponding number as 'unused disk' after running pct rescan.
This has the additional benefit that making snapshots and migration keep working from the PVE WebGUI.

So instead of adding a mountpoint like

mp0: /dev/zvol/tank_ssd/lxc/docker,mp=/var/lib/docker

you then stay within the Proxmox ZFS realm and get a mountpoint like
mp1: local-zfs:vm-101-disk-1,mp=/var/lib/docker

I've created a more detailed guide (tailored to Nextcloud-AIO) here.

Also with Proxmox 7.2 you can run Docker in an unprivileged LXC with only 'nesting' enabled. No need to load any additional kernel modules on the PVE node nor in the LXC.

[techsolo12]

@EcurbNosneb Unfortunately i think you have to do it. When you restore a backup, proxmox take the vm-xxx-disk-1 from the backup and convert it to a subvol-xxx-disk-1. You don't have anymore a RAW VM disk with ext4 after the restore.
So, docker can't use the overlay2 and won't start.

The data on the disk are the same. So, it's the same, if you take the data from your backup on your "/var/lib/docker.backup" or your converted "/rpool/data/subvol-xxx-disk-1"

I wrote a little script to do this automated. It's not the best and should be improved, but since coding is not one of my skills, it is the best i have ever done. :D If anyone had ideas or want to fork it, please fell free to do it!

Can you manually mount the storage? If i understand it right, proxmox cannot can mount the disk.

[EcurbNosneb]

PBS backup method 'stop' successfully backups up /var/lib/docker (zfs+ext vm-110-disk-0) where 'snapshot' failed (fails mount during pbs backup). Unfortunately only 'snapshot' can be used when also doing HA on the container according to the proxmox error message I got.

Since the pbs 'restore' version of /var/lib/docker doesn't recreate the 'ext4' file formatting, I don't see much advantage to worrying more about pbs backups of /var/lib/docker (or why 'snapshot' is not working for me but works for @techsolo12). I'll stick to my rsync backups to the root disk and reuse or recreate the /var/lib/docker disk depending upon the restore situation.

@techsolo12 nice to have your script for creating the zfs+ext4 for restores or for just new docker containers.

[techsolo12]

@EcurbNosneb When you want to use HA with this setup together, you can stop HA via CRON and do the backup. After this, you start HA again. It's all possible through Shell / CLI. It should also be possible to enable a post script after backup to re-enable HA.

I think it should be possible to do your way also automated. But it's more work. Since the LXC should be mounted to copy the data.
Do you stop the docker service before you rsync the docker directory?

[EcurbNosneb]

@techsolo12 I do a cron rsync on /var/lib/docker before each scheduled pbs backup but I don't stop docker. I've done one restore by rsyncing back to /var/lib/docker and it worked fine for me. Docker is not critical for me as I use it to try out various images and if I want to keep one for production, I install it normally in a new or existing LXC. HA is critical for me and so I like to not tinker with it too much. To make PBS backup work as needed I just have a cron rsync with 'backup' toggled off on /var/lib/docker. I prefer the simplicity.

[techsolo12]

Hey Guys. Today i wrote a second script, which can be used to clone a vm-disk to a second LXC. Since the template function is broken with this installation, this is the only semi-automated way to clone a docker container with this installation.

[marconett]

Thanks for the nice tutorial.

I get lost once it gets to the ZFS section. Why do I want to do this? Docker works fine without it. I read related reddit posts and non of those actually explain the reasoning behind this workaround.

Or was the ZFS section needed to make docker run at all the it is now no longer needed with more recent software?
(I found this: https://forum.proxmox.com/threads/lxc-zfs-docker-overlay2-driver.122621/).

docker info:

 Server Version: 24.0.4
 Storage Driver: overlay2
  Backing Filesystem: zfs

proxmox 8.0.3
kernel: 6.2.16-4-pve

[Sieboldianus]

ZFS needs (well, needed) root privileges and these are not available in Docker running in unprivileged LXC containers. Now, there are proceeds to make docker/podman run better in rootless mode, which includes ZFS rootless. This made Docker in unprivileged LXC with ZFS filesystem much easier in the last 1-2 years - there's a good discussion here.

One remaining issue seems to be UID-out-of-range-issues when the docker images are pulled and not build in the unprivileged LXC environment. This last remaining issue is planned to be solved in ZFS 2.2.0, which is currently not available in Proxmox.

I myself still follow the approach described in this blog post without issues, for the last 4 years. I never had a single docker container crash, nor did I experience any corrupted files or other things whatsoever. Personally, I prefer this robustness over any performance gains with the native ZFS approach - which is why I sticked to this 'compromise' for so long. When ZFS 2.2.0 is available, I will see if I migrate my LXC/Docker containers using the native ZFS docker drivers in unprivileged LXC.

tldr: Feel free to try both approaches and see what works best for you.

[SciLor]

I am having trouble using it.

I am on PVE8.
When trying to boot my Docker LXC I get following error:

TASK ERROR: zfs error: cannot open 'rpool/proxmox/container/subvol-2111-disk-4': operation not applicable to datasets of this type

[Sieboldianus]

If you followed the ZFS Volume formatted as XFS approach, your Docker LXC should not be aware of ZFS at all.

[SciLor]

Thank you for the hint. I used pct rescan and used the ressource popping up. But this targets the zfs and so the path is wrong.
Changing it to the absolute /dev/zvol path fixed it.

[ddimick]

I ran into an issue where DNS wasn't available to the docker service for a brief time after boot, maybe 30-60 seconds. Queries would work normally from the root shell. I wasn't able to determine why this was happening. Containers would start, and any DNS queries they would issue would fail for that period of time. I did not encounter this issue when I was running docker in a VM.

For most containers, no big deal, but this would clobber my traefik instance, which relies on a plugin. In this situation, traefik will start, spit some warnings that the plugin middleware wasn't available, and any web services that relied on it wouldn't be available. It would take a manual restart of the traefik container to recover.

As a workaround, I modified docker.service to wait until DNS was working. This was taken from https://unix.stackexchange.com/questions/560529/how-to-start-a-systemd-service-when-local-dns-resolution-is-ready#560703. Use whatever hostname you want in place of one.one.one.one.

/etc/systemd/system/wait-for-dns@.service

[Unit]
Description=Wait for DNS
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'while ! host %i; do sleep 1; done'

/etc/systemd/system/docker.service.d/override.conf

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service containerd.service time-set.target wait-for-dns@one.one.one.one.service
Wants=network-online.target containerd.service wait-for-dns@one.one.one.one.service
Requires=docker.socket