containers/buildah image misses settings for build user

[nolange]

Description

The quay.io/containers/buildah Image seems to miss the configuration
for the non-root build ($HOME/.config/containers/storage.conf).

In turn, running the container as build user will use the default settings and not
work with fuse-overlayfs

Steps to reproduce the issue:

1. Run podman run --rm -it --device /dev/fuse --user build quay.io/containers/buildah buildah info

2. Observe that unlike as root the configuration to use fuse-overlayfs is missing:

       "GraphDriverName": "overlay",
       "GraphOptions": [
           "overlay.imagestore=/var/lib/shared",
           "overlay.mount_program=/usr/bin/fuse-overlayfs",
           "overlay.mountopt=nodev,fsync=0"
       ]

Describe the results you received:

Running as build user, buildah build will not use fuse, and depending on other settings (which I haven't narrowed down),
errors will come up like the one below (apt-get install ca-certificates wont finish):

Unpacking ca-certificates (20230311) ...
dpkg: error processing archive /var/cache/apt/archives/ca-certificates_20230311_all.deb (--unpack):
 unable to install new version of './etc/ca-certificates': Invalid cross-device link

Describe the results you expected:

buildah build proceeding without errors (as it does when not using podman's --user build option)

Output of rpm -q buildah or apt list buildah:

buildah-1.29.1-1.fc37.x86_64

Output of buildah version:

Version:         1.29.1
Go Version:      go1.19.5
Image Spec:      1.0.2-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        1.0.0
libcni Version:  v1.1.2
image Version:   5.24.1
Git Commit:      
Built:           Fri Feb 17 10:05:41 2023
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Output of podman version if reporting a podman build issue:

Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
Go Version:   go1.19.6
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of cat /etc/*release:

NAME="Fedora Linux"
VERSION="37 (Container Image)"
ID=fedora
VERSION_ID=37
VERSION_CODENAME=""
PLATFORM_ID="platform:f37"
PRETTY_NAME="Fedora Linux 37 (Container Image)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:37"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f37/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=37
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=37
SUPPORT_END=2023-11-14
VARIANT="Container Image"
VARIANT_ID=container

Output of uname -a:

Linux ac677b1df0a6 6.1.0-6-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.15-1 (2023-03-05) x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:
(comments removed)

[storage]
driver = "overlay"
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"

[storage.options]

additionalimagestores = [
"/var/lib/shared",
]

pull_options = {enable_partial_images = "false", use_hard_links = "false", ostree_repos=""}

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"
mountopt = "nodev,fsync=0"
[storage.options.thinpool]

[rhatdan]

We want to fall back to fuse-overlayfs not use it by default. If the kernel supports using native overlay for the rootless user we should use it, If we hard code fuse-overlayfs into the configuration, then we won't be able to use native overlay.

Podman/Buildah are supposed to be smart enough to fall back to fuse-overlayfs (if it is installed) and /dev/fuse device exists and native overlayfs is not supported.

[nolange]

We want to fall back to fuse-overlayfs not use it by default. If the kernel supports using native overlay for the rootless user we should use it, If we hard code fuse-overlayfs into the configuration, then we won't be able to use native overlay.

AFAIU the docker image doesnt change the defaults, but adds lines to use a custom mount step:

buildah/contrib/buildahimage/Containerfile

Lines 70 to 74 in a8ba52d

	RUN sed -e 's|^#mount_program|mount_program|g' \
	-e '/additionalimage.*/a "/var/lib/shared",' \
	-e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' \
	/usr/share/containers/storage.conf \
	> /etc/containers/storage.conf && \

Podman/Buildah are supposed to be smart enough to fall back to fuse-overlayfs (if it is installed) and /dev/fuse device exists and native overlayfs is not supported.

Your own docker image sets that config - but only for root?
Running buildah in a rootless container seems a bit more involved, and it doesn't work for me (when run as user). I dont understand how the Invalid cross-device link crops up, but I suspect its the missing configuration when run as build user.

[rhatdan]

This is a bug
-e 's|^#mount_program|mount_program|g' \

We should not be turning on the mount_program for rootful mode, we should use native overlay.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc you reverted my fix for this, could you followup on this?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[flouthoc]

@flouthoc you reverted my fix for this, could you followup on this?

Yes let me check this.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[flouthoc]

Checking this now.

[flouthoc]

@nolange PR #4905 should close this and long term kernel agnostic fix should go in c/storage I think.