python installation issues on docker snap

[greatquux]

Install docker snap and create an Ubuntu 22.10 container, and try to install python, you will get an error:
Setting up python3.10-minimal (3.10.7-1) ...
[Errno 13] Permission denied: '/usr/lib/python3.10/pycache/future.cpython-310.pyc.139786761293248'dpkg: error processing package python3.10-minimal (--configure):
installed python3.10-minimal package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
python3.10-minimal
E: Sub-process /usr/bin/dpkg returned an error code (1)

See
https://forums.docker.com/t/bug-on-apt-install-permission-denied/100196/6
for discussion. Apparently only a problem with the snap version.

[greatquux]

BTW once I did find where my daemon.json file was, I switched to using the vfs engine and did not have any further problems with python. But I think the issue should be fixed for overlay2.

[mgajda]

The problem still persists, and Docker recommends installing from .deb package repository. This is workaround recommended by StackOverflow.

[plars]

For now, you can work around this by using the vfs storage-driver instead:

• edit /var/snap/docker/current/config/daemon.json and replace "overlay2" with "vfs" for the storage-driver
• run sudo snap restart docker

[kurapov-peter]

Observing the same issue here

[zhsj]

The issue can be easily triggered with following command

docker run --rm ubuntu:devel sh -c 'mkdir -p /usr/lib/python3/__pycache__; echo 1 > /usr/lib/python3/__pycache__/x.pyc.1'

This seems to caused by the apparmor policy.

Maybe this one?

https://github.com/snapcore/snapd/blob/19c123f2b3f67c6e79e1f0a2af7c0721194b3a70/interfaces/apparmor/template.go#L108

deny /usr/lib/python3*/{,**/}__pycache__/**.pyc.[0-9]* w,

But I'm out of idea why the kernel doesn't use the path which is translated after overlayfs.

[zhsj]

Mar 20 18:27:58 lunar-vm2 kernel: audit: type=1400 audit(1679336878.729:17246): apparmor="DENIED" operation="open" class="file" profile="snap.docker.dockerd" name="/usr/lib/python3/__pycache__/x.pyc.1" pid=1629 comm="sh" requested_mask="w" denied_mask="w" fsuid=0 ouid=0

Looks like it's indeed blocked by the snap apparmor.

My guess is that the realChroot patch https://github.com/docker-snap/docker-snap/blob/8e57d1fdee6d4b31f9a2f389c6081451d8bd3191/dockerd-patches/snappy-real-chroot.patch#L9-L14 doesn't work now. Maybe it's broken since docker was split to containerd and runc?

[zhsj]

Hmm, this issue doesn't present on ubuntu 20.04. So I fail to add a regression test in GitHub actions.

[zhsj]

It happens when I use 5.19 kernel (the HWE version) in Ubuntu 22.04.

[zhsj]

So I have a patch (#116) to workaround it. But it looks dangerous.

After #116, snap.docker.dockerd profile is applied to the whole path (like /var/snap/docker/common/var-lib-docker/overlay2/...).
But docker-default profile is also applied to that.

Compared to the old behavior on kernel 5.15 (without the patch), snap.docker.dockerd profile is applied to whole path, while docker-default profile is applied to the path in container (like /usr/lib/python3/__pycache__/...)

@xnox do you have some thoughts wrt the kernel?

[xnox]

In recent kernels there is overlay mitigations available so it should be possible to make this work. Will need to check this with apparmor maintainer

[zhsj]

So far I see it fails from Ubuntu kernel 5.19.0-1001.1, working on 5.18.0-1001.1.

However I didn't find any working Debian kernel for this snap :/ (old versions fail with cgroup/bpf thing, new versions fail for this issue).

So my guess is that the docker-snap only works with some Ubuntu specific overlayfs patches, which is lost since Ubuntu kernel 5.19.

[xnox]

So far I see it fails from Ubuntu kernel 5.19.0-1001.1, working on 5.18.0-1001.1.

However I didn't find any working Debian kernel for this snap :/ (old versions fail with cgroup/bpf thing, new versions fail for this issue).

So my guess is that the docker-snap only works with some Ubuntu specific overlayfs patches, which is lost since Ubuntu kernel 5.19.

we do have some feature changes and regressions in overlayfs & shiftfs which both might be at play here.

This is very much annoying.

[xnox]

with snapd built with canonical/snapd#12822 this seems to work for me now. Waiting to see if snapd team will agree on the approach.