Unable to mount squashfs via singularity exec with kernel 5.4.0

[gemarcano]

Version of Singularity:

$ singularity version
3.5.0

Expected behavior

Success when running:

$ singularity exec  build/dev.simg bash

Actual behavior

On Linux kernel 5.3.12, singularity works as intended for some reason. on 5.4.0, I get the following error:

$ singularity exec  build/dev.simg bash
FATAL:   container creation failed: mount /proc/self/fd/3->/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: kernel reported a bad superblock for squashfs image partition, possible causes are that your kernel doesn't support the compression algorithm or the image is corrupted

Furthermore, dmesg logs the following (once per attempt):

[ 3878.270562] squashfs: Unknown parameter 'errors'

Steps to reproduce this behavior

Change kernel to 5.4.0 and reboot with said kernel, then attempt to call 'singularity exec' with a prebuilt image.

What OS/distro are you running

Running Gentoo Linux unstable (~amd64), Linux kernel 5.4.0. Desktop environment is Plasma.

How did you install Singularity

Built from source using ebuild from https://bugs.gentoo.org/694048 modified to use version 3.5.0 instead of 3.4.0 (literally renaming the ebuild from singularity-3.4.0.ebuild to singularity-3.5.0.ebuild). After installing the ebuild in my local portage repository, all I had to do was

# emerge singularity

and wait until compilation and installation completed.

Additional information/comments

I will stress that the exact same binary works with kernel version 5.3.12, but not with 5.4.0 (using GRUB to switch between kernels on reboot for testing). After diving through the kernel source, I'm not sure why 5.3.12 even works, honestly, but the reason it doesn't work on 5.4.0 is that in fs/fs_context.c in the kernel tree squashfs does not provide an argument parser, so the errors argument cannot be parsed and the function vfs_parse_fs_param eventually returns an EINVAL as a result, along with the error message being reported in the dmesg.

Arguably both the kernel and singularity are wrong here (I think)-- the kernel is wrong because this seems like a userspace regression, and singularity is wrong by passing an argument for mounting that is not supported or understood by squashfs.

I will be away on travel for about a week, but let me know if you need anything else from me to debug this.

[cclerget]

Hi @gemarcano , thanks for the details, maybe a side effect of some changes at the new mount API, afaik squashfs never took any filesystem specific options and those options was just ignored until now :). Will fix that. Thanks !

[gemarcano]

I can confirm that version 3.5.1-rc.1 works, I'm assuming the work you merged fixed it. Thanks!