Checks for /boot/EFI write permissions are cached for too long

[mkszuba]

Describe the bug

If the fwupd daemon is started on a system with /boot mounted read-only, it keeps thinking it remains read-only - and refuse writing to /boot/EFI/ - even after the partition having been remounted rw. It is necessary to kill the daemon for the change to take effect.

Steps to Reproduce

# killall fwupd
# mount /boot -o remount,ro
# fwupdmgr update
[...]
╔══════════════════════════════════════════════════════════════════════════════╗
║ Upgrade System Firmware from X to Y?                                         ║
╠══════════════════════════════════════════════════════════════════════════════╣
║ The LVFS runs tests on certain types of firmware to check they are           ║
║ valid.Some tests are still pending and will be completed shortly.You can     ║
║ refresh this page to find out when the firmware has been tested.             ║
║                                                                              ║
║ [REDACTED]  must remain plugged into a power source for the duration of the  ║
║ update to avoid damage.                                                      ║
╚══════════════════════════════════════════════════════════════════════════════╝
Perform operation? [Y|n]: y
Decompressing…           [                                       ] Less than one minute remaining…
Failed to create '/boot/EFI/gentoo/fw': Read-only file system
# mount /boot -o remount,rw
# fwupdmgr update
[...]
╔══════════════════════════════════════════════════════════════════════════════╗
║ Upgrade System Firmware from X to Y?                                         ║
╠══════════════════════════════════════════════════════════════════════════════╣
║ The LVFS runs tests on certain types of firmware to check they are           ║
║ valid.Some tests are still pending and will be completed shortly.You can     ║
║ refresh this page to find out when the firmware has been tested.             ║
║                                                                              ║
║ [REDACTED]  must remain plugged into a power source for the duration of the  ║
║ update to avoid damage.                                                      ║
╚══════════════════════════════════════════════════════════════════════════════╝
Perform operation? [Y|n]: y
Decompressing…           [                                       ] Less than one minute remaining…
Failed to create '/boot/EFI/gentoo/fw': Read-only file system
# killall fwupd
# fwupdmgr update
[...]
╔══════════════════════════════════════════════════════════════════════════════╗
║ Upgrade System Firmware from X to Y?                                         ║
╠══════════════════════════════════════════════════════════════════════════════╣
║ The LVFS runs tests on certain types of firmware to check they are           ║
║ valid.Some tests are still pending and will be completed shortly.You can     ║
║ refresh this page to find out when the firmware has been tested.             ║
║                                                                              ║
║ [REDACTED]  must remain plugged into a power source for the duration of the  ║
║ update to avoid damage.                                                      ║
╚══════════════════════════════════════════════════════════════════════════════╝
Perform operation? [Y|n]: y
Waiting…                 [***************************************] Less than one minute remaining…
Successfully installed firmware
Do not turn off your computer or remove the AC adapter while the update is in progress.
An update requires a reboot to complete. Restart now? [y|N]:

Expected behavior

Operations requiring write access to /boot/EFI begin to succeed as soon as the file system in question has been remounted read-write. Or possibly fwupd taking care of remounting /boot rw (and back as ro afterwards) itself.

fwupd version information

All versions from at least 1.2.4 (see https://bugs.gentoo.org/677352) up to and including 1.9.3, installed using Gentoo Linux ebuilds in either case. Current Git head not tested.

**fwupd device information**

Likely not relevant due to the issue having been observed on several different systems, and therefore omitted for privacy reasons.

Additional questions

• Operating system and version: Gentoo Linux
• Have you tried rebooting? Yes, counless times since the issue was first observed.
• Is this a regression? Given now long it has been around, not really.

[hughsie]

I don't know why this happens:

mount /boot -o remount,rw
fwupdmgr update
Failed to create '/boot/EFI/gentoo/fw': Read-only file system

fwupd isn't actually generating the Read-only file system error -- we're not checking for R/O at all. Maybe it's because fwupd has an open fd on something in /boot and it's still effectively r/o? Although that would be pretty weird. Maybe you can use something like ls -l /proc/<fwupd_pid>/fd | wc - l or fstat to see if fwupd has an open fd?

Also, do you know why this bug might be Gentoo-specific? We've not seen this before.

[superm1]

I don't know why this happens:

mount /boot -o remount,rw
fwupdmgr update
Failed to create '/boot/EFI/gentoo/fw': Read-only file system

fwupd isn't actually generating the Read-only file system error -- we're not checking for R/O at all. Maybe it's because fwupd has an open fd on something in /boot and it's still effectively r/o? Although that would be pretty weird. Maybe you can use something like ls -l /proc/<fwupd_pid>/fd | wc - l or fstat to see if fwupd has an open fd?

Also, do you know why this bug might be Gentoo-specific? We've not seen this before.

Could it be because of the systemd unit?

[hughsie]

Could it be because of the systemd unit?

I wondered that too, but that wouldn't explain why restarting it make it suddenly r/w -- no?

[superm1]

Does this also fail?

# mount /boot -o remount,ro
# killall fwupd
# fwupdmgr get-devices
# mount /boot -o remount,rw
# fwupdmgr update

[superm1]

I wondered that too, but that wouldn't explain why restarting it make it suddenly r/w -- no?

yeah good point. My next guess here would be that glib internally doesn't release the fd for a given path specifically in the error path, but I looked through glib code and I didn't see anywhere it should cache an fd. It just ends up calling mkdir with the path.

[mkszuba]

Apologies for the delay responding, holiday season. Anyway:

Maybe you can use something like ls -l /proc/<fwupd_pid>/fd | wc - l or fstat to see if fwupd has an open fd?

Good idea, I'll keep that in mind for the next time I have an update to install on one of the affected systems.

Also, do you know why this bug might be Gentoo-specific? We've not seen this before.

I do not know if that affects Gentoo only, it's just that all of my "/boot is mounted ro" systems happen to run Gentoo.

Could it be because of the systemd unit?

If it is then it is likely not a Gentoo-specific thing, as we do not customise fwupd units.

Does this also fail?

I've tried that before and yes, it does. Which makes sense because my systems do not start fwupd at either boot or login time, it only gets launched when fwupdmgr is run.

[superm1]

Apologies for the delay responding, holiday season. Anyway:

Maybe you can use something like ls -l /proc/<fwupd_pid>/fd | wc - l or fstat to see if fwupd has an open fd?

Good idea, I'll keep that in mind for the next time I have an update to install on one of the affected systems.

Also, do you know why this bug might be Gentoo-specific? We've not seen this before.

I do not know if that affects Gentoo only, it's just that all of my "/boot is mounted ro" systems happen to run Gentoo.

Could it be because of the systemd unit?

If it is then it is likely not a Gentoo-specific thing, as we do not customise fwupd units.

Does this also fail?

I've tried that before and yes, it does. Which makes sense because my systems do not start fwupd at either boot or login time, it only gets launched when fwupdmgr is run.

You should be able to use the reinstall command to trigger the same behavior.

[mkszuba]

No open descriptors to anywhere in /boot, or indeed nowhere on the file system except /dev/null and /var/lib/fwupd/pending.db.

I am beginning to suspect this has got something to do with the way systemd sets up filesystem namespaces for protected services like fwupd.service. No idea if this is something that could be addressed on the fwupd side, or indeed at all, though.

[hughsie]

Could you ask on the systemd mailing list perhaps?

[superm1]

What if you run the daemon manually without systemd? Does it still happen?

[mkszuba]

What if you run the daemon manually without systemd? Does it still happen?

No, with the daemon having been launched manually remounting /boot rw takes immediate effect.

I have just e-mailed systemd-devel about this, let's see what they say.

[superm1]

What if you run the daemon manually without systemd? Does it still happen?

No, with the daemon having been launched manually remounting /boot rw takes immediate effect.

I have just e-mailed systemd-devel about this, let's see what they say.

Great thanks! It's good to know it's caused by systemd. Can you get a ticket opened in their tracker? Then we can close the fwupd one as a duplicate and track there.