Proxmox 7.3.4 LXC Container with docker

[netsrot303]

Since the last Proxmox update (7.3.4) I have problems with the LXC container with Docker. In the LXC container the created docker containers do not run anymore. Does anyone else have these problems? In the forum I read this:
https://forum.proxmox.com/threads/todays-kernel-firmware-update-has-really-messed-up-my-boxes.119933/#post-520896

[tteck]

What file system are you running?

[netsrot303]

On Proxmox host runs zfs

[tteck]

I run ext4 so I can't troubleshoot. Did you try fuse-overlayfs like recommended in the post you referred to?

[netsrot303]

This fix worked for me only with a privileged container.
I'll test your new script in a moment.

[tteck]

If you'd like to test Docker with the option to use fuse_overlayfs (Advanced)

bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/ct/docker-v4.sh)"

@lordratner

[netsrot303]

The install scrippt fails.
++ curl -sSL https://get.docker.com
++ EXIT=100
++ LINE=116
++ error_exit
++ trap - ERR
++ local 'reason=Unknown failure occurred.'
++ local 'msg=Unknown failure occurred.'
++ local 'flag=\033[01;31m‼ ERROR \033[m100@116'
++ echo -e '\033[01;31m‼ ERROR \033[m100@116 Unknown failure occurred.'
‼ ERROR 100@116 Unknown failure occurred.
++ exit 100

[tteck]

OK, made some changes. If it works, check (docker info | grep "Storage Driver") to make sure it's using fuse-overlayfs as the driver

[netsrot303]

The install script fails on installing docker. I tested privileged and unprivileged, both fails on installing docker

[tteck]

Sorry, I've been making changes every minute or so. I just made my last change for awhile, installs on my end now. Leave me your results and I'll get back on it (if needed) later. Thank you for helping me find a solution!

[netsrot303]

Now the install script finished and fuse-overlyfs is set as driver. portainer is running too. "docker pull jc21/nginx-proxy-manager" fails inside the lxc container with this error message "failed to register layer: ApplyLayer exit status 1 stdout: stderr: unlinkat /var/cache/debconf: invalid argument". I tested this with an unprivileged Container. Later i will test this with a privileged LXC Container.

[tteck]

I guess we can call it some progress 😜

[netsrot303]

So, I have now done the same with a privileged container and there I was able to pull the docker image from the NGINXproxymanager. The problem is only with an unprivileged container that you can't create some docker containers.

[tteck]

What kernel are you using?

[netsrot303]

Linux pve 5.15.83-1-pve #1 SMP PVE 5.15.83-1 (2022-12-15T00:00Z) x86_64 GNU/Linux

[tteck]

I've been using (uname -r) 6.1.0-1-pve I'm wondering if people running newer kernels are having this issue?

https://forum.proxmox.com/threads/opt-in-linux-6-1-kernel-for-proxmox-ve-7-x-available.119483/

I'm trying to narrow down what PVE 7.3-4 changed to cause this. kernel? zfs? ect... If we knew that, I'm guessing that a permanent fix would be much easier.

[rallegade]

Well i can tell you that much that it absolutely does use overlay2 after a fresh install with the script, which for me at least is causing issues with docker.
As soon as i change the driver to fuse-overlayfs in /etc/docker/daemon.json the problem is fixed.
Added some pics of it.

Output of docker info | grep "Storage Driver" straight after setup with the script

Resulting in the following error

after changing to fuse-overlayfs in /etc/docker/daemon.json

so to me the easiest fix seems to be forcing docker to use fuse-overlayfs by adding it to /etc/docker/daemon.json

[tteck]

Would you test the dev Docker LXC now?

[rallegade]

sure

[rallegade]

[rallegade]

seems like you have a typo in the /etc/docker/daemon.json there is missing a ,

it should be as the following

{
  "log-driver": "journald",
  "storage-driver": "fuse-overlayfs"
}

[netsrot303]

should i try the dev branch or the main branch?

[tteck]

either, dev is Ubuntu main is Debian
everything else is the same

[netsrot303]

ah okay, so for both only privileged containers works?

[lordratner]

OK, so I just spun up a V4 prod (debian) LXC. Unprivileged, fuse-overlayfs. It's all working great. I'm only running SWAG and Authelia on it for now, but no issues.

[netsrot303]

On host system you have zfs? can you pull the docker image "jc21/nginx-proxy-manager:latest" without errors?

[rallegade]

I was able to yes

[lordratner]

On host system you have zfs? can you pull the docker image "jc21/nginx-proxy-manager:latest" without errors?

Yes, I'm using zfs. I pulled SWAG, which includes nginx. There were no issues