Podman build in gitlab runner docker container cannot access overlay

[alba02119451]

I will preface this post by saying that there are multiple factors (podman, docker, gitlab, and Linux hardening) that could be at play here, so if I should be asking this elsewhere, or you know of any information I could provide to shine a light onto the issue, please do let me know.

I am porting a project from one GitLab instance to another and I am running into some issues. The image is built as follows:

FROM docker.io/hashicorp/terraform:1.15.5 AS terraform
FROM docker.io/alpine/ansible:2.20.0
COPY --from=terraform /bin/terraform /usr/bin/terraform
RUN apk add --no-cache jq envsubst moreutils ansible-lint

The build command is simply podman build -f Dockerfile followed by annotations, with no other modifications.

We have a custom image that we build on our self-hosted GitLab instance. Our runners use podman to run jobs within containers, while the runners themselves are installed on the (virtual) machine itself. On the GitLab instance where it was originally developed, this works fine, but porting it to the other instance proves problematic...

The other instance uses docker for their container runners, and runs the GitLab runner itself also as a docker container. On this GitLab instance, building the image results in the following error:

Error: mounting new container: mounting build container "addd07e8d25b6155b39c0f5d3f770415da2197722ae5e7671edc9497a7891df6": creating overlay mount to /var/lib/containers/storage/overlay/fd6a74fabe13121ed68d4f28fbbd6c88f8ab3e37d37a2d5223684cd4db999e9a/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/6TH4U7ZRCKBCQJCW3WVJCWHKX3:/var/lib/containers/storage/overlay/l/HBWWH22NSTP2OJXLDCQ3AGYYMP:/var/lib/containers/storage/overlay/l/N47UM73J4SXWGRDOUZ4F4PVJE4:/var/lib/containers/storage/overlay/l/VQWDWPX2FRJBP7UM5TH7EPVCG7,upperdir=/var/lib/containers/storage/overlay/fd6a74fabe13121ed68d4f28fbbd6c88f8ab3e37d37a2d5223684cd4db999e9a/diff,workdir=/var/lib/containers/storage/overlay/fd6a74fabe13121ed68d4f28fbbd6c88f8ab3e37d37a2d5223684cd4db999e9a/work,nodev,fsync=0,volatile": using mount program /usr/bin/fuse-overlayfs: fuse: device /dev/fuse not found. Kernel module not loaded?
fuse-overlayfs: cannot mount: No such file or directory
: exit status 1

Looking around on forums and documentation, I found that, in order to build/run rootless containers, you need an overlay implementation active on the host, with various options listed. The strange thing is that the host on which this runs does have an overlay implementation active (on XFS iirc), yet the builder cannot seem to access it properly. I have tried to run it with --cap-add SYS_ADMIN as well in case it was a permission issue, but to no avail.

I was not able to find a lot of documentation or discussion on this, so I decided to come here. Does anyone here have an idea as to why podman cannot access the necessary overlay module? Any input, tip, or suggestion is welcome, and I will do my best to provide information that is requested.

[nalind]

fuse-overlayfs is what we still configure in the image's copy of /etc/containers/storage.conf, and FUSE-based filesystems depend on having a usable /dev/fuse node in the container. If your kernel is recent enough, though, it should be possible to use regular overlay, without fuse-overlayfs. If you know you have the real thing available, commenting out the mount_program line in the file before running buildah itself in the container, and possibly editing the mountopt setting, should switch what the copy of buildah in the container attempts to use.

[alba02119451]

Many thanks for your help! Commenting out mount_program and setting mountopt to just nodev did the trick.

To any future people that might stumble upon this, I'll also link the documentation that I found based off of this response:
https://github.com/containers/storage/blob/main/docs/containers-storage.conf.5.md
https://www.man7.org/linux/man-pages/man8/mount.8.html