linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[rdslw]

Is this a new report?

Yes

System Info

Void x86_musl

Package(s) Affected

linux5.15-5.15.*

Does a report exist for this bug with the project's home (upstream) and/or another distro?

no, this is void specific bug.

Expected behaviour

package linux5.15 is able to install even if /boot is a vfat partition, and chown/chmod commands may return errors, which are ignored by install scripts of linux5 package

Actual behaviour

If the /boot (where /vmlinuz-XXX and config-XXX and initramfs-XXX are installed) is vfat filesystem (not the ext4/btrfs/others), installation of linux5.15 package fails, due to the inability to set ownership and permissions during xbps install.

What's funny, files ARE copied to /boot, but the install process fails with error after copy. So it is enough to run xpbs install three times in a row (without any fixes) and the third run, completes fine.

Important: this bug is old, it was not introduced recently, but probably been with us for a long time.

Error messages:

linux5.15-5.15.131_1: updating to 5.15.137_1 ...
linux5.15-5.15.137_1: unpacking ...
ERROR: linux5.15-5.15.137_1: [unpack] failed to extract file `./boot/vmlinuz-5.15.137_1': Operation not permitted
ERROR: linux5.15-5.15.137_1: [unpack] failed to extract files: Operation not permitted
ERROR: linux5.15-5.15.137_1: [unpack] failed to unpack files from archive: Operation not permitted

linux5.15-5.15.131_1: updating to 5.15.137_1 ...
linux5.15-5.15.137_1: unpacking ...
ERROR: linux5.15-5.15.137_1: [unpack] failed to extract file `./boot/config-5.15.137_1': Operation not permitted
ERROR: linux5.15-5.15.137_1: [unpack] failed to extract files: Operation not permitted
ERROR: linux5.15-5.15.137_1: [unpack] failed to unpack files from archive: Operation not permitted

Steps to reproduce

1. use /boot on vfat partition (for examply to have shared EFI and /boot space).
2. try to upgrade (important) linux5.15 package (newer kernel)

[Duncaen]

I don't think that is the issue, /boot on vfat for efi is a pretty standard setup. Can you maybe strace this?

[rdslw]

@Duncaen sure, I can. Here I was using 'xbps-install -Su' command, would you like to just strace it, and try to find calls around error/post log?

Currently I'm only able to downgrade linux kernel package, as I have the newest one installed today. Can you guide me how to do it, as simple xbps-install --force --debug /var/cache/xbps/linux5.15-5.15.131_1.x86_64-musl.xbps results in error: Package '/var/cache/xbps/linux5.15-5.15.131_1.x86_64-musl.xbps' not found in repository pool

P.S. My guess about chmod/chown is because after each command/error, the offending file from error message is unpacked to /boot, but the whole command aborts while the next (exactly the same xbps-install -Su) invocation moves forward and fails on another file, then third call succeeds. I might be wrong though.

[classabbyamp]

don't use the full path, just the package name

[rdslw]

@classabbyamp appreciate you want to help. Unfortunately, it's not the path.
For reference the proper procedure requires two separate commands:

xbps-rindex -a /var/cache/xbps/pkg-1.0_1.xbps
xbps-install -R /var/cache/xbps/ -f pkg-1.0_1

as explained here: https://docs.voidlinux.org/xbps/advanced-usage.html?highlight=xdowng#via-xdowngrade

strace results for this issue follows shortly in my next comment.

[rdslw]

@Duncaen as I expected, this is due to the chown, strace is:

[...]
open("boot/vmlinuz-5.15.131_1", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0644) = 7
fcntl(7, F_SETFD, FD_CLOEXEC)           = 0
fcntl(7, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
write(7, "MZ\352\7\0\300\7\214\310\216\330\216\300\216\3201\344\373\374\276@\0\254 \300t\t\264\16\273\7\0"..., 13312) = 13312
read(6, "ze\240e}M\205\363\203\362\304\214`f\261\306M=\354\340J\273\272Y\247E\22\343+\323+\212"..., 4096) = 4096

[... lot of writing to FD no 7]
write(7, "\20\2315\fQ\23\f\306\321\226!Z\331*D\333\333\206h}\273\20\235\302\304\366!\272\211\316\35B"..., 102176) = 102176
fchown(7, 0, 0)                         = -1 EPERM (Operation not permitted)
lchown("boot/vmlinuz-5.15.131_1", 0, 0) = -1 EPERM (Operation not permitted)
utimensat(7, NULL, [{tv_sec=1698864954, tv_nsec=0} /* 2023-11-01T19:55:54+0100 */, {tv_sec=1694612722, tv_nsec=0} /* 2023-09-13T15:45:22+0200 */], 0) = 0
close(7)                                = 0
writev(2, [{iov_base="", iov_len=0}, {iov_base="ERROR: ", iov_len=7}], 2ERROR: ) = 7
writev(2, [{iov_base="", iov_len=0}, {iov_base="linux5.15-5.15.131_1: [unpack] f"..., iov_len=106}], 2linux5.15-5.15.131_1: [unpack] failed to extract file `./boot/vmlinuz-5.15.131_1': Operation not permitted) = 106
writev(2, [{iov_base="\n", iov_len=1}, {iov_base=NULL, iov_len=0}], 2
) = 1
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 7
connect(7, {sa_family=AF_UNIX, sun_path="/dev/log"}, 12) = 0
sendto(7, "<11>Nov  1 18:56:00 xbps-install"..., 141, 0, NULL, 0) = 141
writev(2, [{iov_base="", iov_len=0}, {iov_base="ERROR: ", iov_len=7}], 2ERROR: ) = 7
writev(2, [{iov_base="linux5.15-5.15.131_1: [unpack] f"..., iov_len=80}, {iov_base=NULL, iov_len=0}], 2linux5.15-5.15.131_1: [unpack] failed to extract files: Operation not permitted
) = 80
sendto(7, "<11>Nov  1 18:56:00 xbps-install"..., 114, 0, NULL, 0) = 114
writev(2, [{iov_base="", iov_len=0}, {iov_base="ERROR: ", iov_len=7}], 2ERROR: ) = 7
writev(2, [{iov_base="", iov_len=0}, {iov_base="linux5.15-5.15.131_1: [unpack] f"..., iov_len=91}], 2linux5.15-5.15.131_1: [unpack] failed to unpack files from archive: Operation not permitted) = 91
writev(2, [{iov_base="\n", iov_len=1}, {iov_base=NULL, iov_len=0}], 2
) = 1
sendto(7, "<11>Nov  1 18:56:00 xbps-install"..., 126, 0, NULL, 0) = 126
close(6)                                = 0
munmap(0x79b456975000, 4591616)         = 0
umask(022)                              = 022
writev(2, [{iov_base="", iov_len=0}, {iov_base="Transaction failed! see above fo"..., iov_len=42}], 2Transaction failed! see above for errors.
) = 42
fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_CUR, l_start=0, l_len=0}) = 0
close(3)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++

[Duncaen]

I wonder if this is either because of a specific kernel version or different mount flags than everyone else uses. This works fine for the raspberry pi images which have vfat /boot by default. And for other x86_64 systems, like the one I'm currently writing from where it works fine too.

[Duncaen]

Correctly handling this in xbps would require quite a bit of rework since currently libarchive extracts, handles the mode, and file times. xbps would have to manually chown the files and then I guess ignore chown errors on vfat, so that EPERM from open/write don't get ignored.

[classabbyamp]

what's the output of cat /etc/fstab and mount?

[rdslw]

@Duncaen @classabbyamp I do use different /boot mount flags, so this might be the problem. Let me know any tests you want to do it.

$> grep boot /etc/fstab                                                                                                               ~
LABEL=ESP           /boot         vfat          rw,relatime,fmask=0117,dmask=0007,gid=10,iocharset=iso8859-1,codepage=437,shortname=mixed,errors=remount-ro    0 2
$> mount |grep boot                                                                                                                   ~
/dev/nvme0n1p1 on /boot type vfat (rw,relatime,gid=10,fmask=0117,dmask=0007,allow_utime=0020,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro)

[Duncaen]

Weird mine are effectively the same, except gid. The quiet option should be what would make attribute changes return 0 instead of errors, but I don't even have that set here so not sure why its returning 0 for me anyways. Maybe I'm missing some other place or reason why errors are ignored.

Maybe its the gid, but xbps-install is running as root so it shouldn't care. Oh wait its chown that failing and not chmod, so it does make sense that this fails because you set the gid to 10 instead of root (0).

[rdslw]

Doing tests, appreciate your help @Duncaen
started with quiet while leaving gid=10 in place. Manual upgrade of package did NOT result in error. Will wait to be sure, for next official update of linux5.15 and normal xbps-install -Su and report here if quiet solved it.

[ahesford]

Seems like this is caused by specific configuration options and the errors are not unexpected. If you have convincing evidence that this issue is not caused by mount options, please let us know.

[rdslw]

I can confirm that adding quiet to the mount options of /boot resolved problem.
It is still bug in xbps, as it should correctly be able to write linux5.15 files on the vfat partition (this is widely used scenario) mounted with gid option, nevertheless, adding quiet is easier (especially if user previously added gid himself :)

[Duncaen]

The package contains files that are owned by root, xbps tries to set the owner/group. I don't think xbps should ignore failures of setting the expected owner, even making this filesystem specific seems wrong. What is the reason to set gid= on /boot and what user is that?

[rdslw]

Thanks for answering it!

The actual reason is to have ability to writeable (some) files on /boot vfat partition, while not allowing all users to write them. Here gid points to some wheels/adm gid which only some users belong. Due to the limitations of vfat fs, it's widely adopted to use some form of gid/uid/fmask/dmask on such fs, to have simple access control.

(Just to make it clear: xbps is being run under root)

But moving back to the error:
I think "xbps should ignore failures" is not true, as there is no ownership on vfat filesystem type (used on /boot) therefore this is not a failure, but xbps bug (not understanding that sometimes this operation (permissions/ownership) is not needed)

I consider FS settings machine, not void specific. One can ask, why using vfat on this filesystem? ;)