systemd: Do not unmount /storage and /flash at shutdown as this always fails #546
Conversation
|
Could this be something we backport to LE 7.0 as a bugfix? |
|
If there are plans for any further LE 7.0 releases then I can backport this based on whatever we end up merging to master. |
|
Meant to ping @stefansaraev not @seo. :) |
|
I have never seen this issue, but it seems to be a race condition due to wrong service ordering OE/LE has too much services with DefaultDependencies=no declared. this is a bad thing, and behaviour may suddenly change every time you update systemd, or you add yet another service. my 2 cents. all services that use DefaultDependencies=no should be looked at, and whenever possible, cleaned up better sooner than later. those that run on shutdown should be Before=shutdown.target and WantedBy=shutdown.target (or poweroff/reboot targets). those should also always have Type=oneshot and RemainAfterExit=yes less services that break default dependency chain - less trouble in future. I lost track on what's going on with OE/LE development, my very stripped down fork has exactly 5 non-systemd shipped services, so perhaps I should not comment too much. of course, if it is really not possible to safely umount /flash and /storage, this PR is ok. dirty but ok. EDIT:
true, but systems with > 1G of ram will run from ram (/dev/SYSTEM), there you can even umount /flash while LE is runing, and/or do "in place upgrade" replacing contents of /flash and just rebooting, I have done that for years on my old ION :) EDIT: oh well, and long time ago, I voted against adding shutdown.sh support. but that's another story ;) |
|
@stefansaraev thanks, your comments sound reasonable and perhaps it's something we can look at in the future to try and improve the way we use systemd (though likely not something for the feint of heart!) |
|
offtopic. umounting storage if it's on nfs should be fine if you use KillSignal=SIGKILL in connman.servce /me runs |
|
i got this same issue from some time ... |
|
Same here, haswell NUC. With this PR there's no failed unmounting during shutdown, but the console is still spammed with OK messages. |
Presumably the spam is caused by a different service in your case? Diagnosing in this PR wouldn't be the right place but try adding a |
|
my 2 cents, I'm getting the console spam on shutdown/reboot running LE on NUC, error is Failed unmouting /storage |
|
@jester-xbmc if you want to test this it's in my last Generic test build: http://forum.kodi.tv/showthread.php?tid=269815&pid=2376765#pid2376765 |
|
@MilhouseVH I know ;-) thx |
|
Good to go? |
|
I think so. Easy enough to revert if we do find a problem (or better solution). |
The default
systemd-halt.servicewill unmount all mounted file systems prior to shutting down the system.Unfortunately, this means that systemd attempts to unmount
/storageearly in the shutdown process when/storageis still in use (for various purposes, most notably the system journal, plus/storage/.configas we haven't executed any shutdown scripts yet...), and consequently the unmount of/storagealways fails.Here is the shutdown log based on current master when booting Generic on i5 NUC - note that I have two additional mount points
/storage/freenasand/storage/datathat are being correctly unmounted:(Full log: http://sprunge.us/CWLD)
This failure to unmount
/storageresults in the following user experience on shutdown:With this PR, we prevent the umount.target from unmounting
/storagein umount.c - we don't want it unmounted when we still need it, and it's still in use anyway so unmounting will always fail.Following this PR, the
umount.targetis a success on shutdown:(Full log: http://sprunge.us/OEAA)
And the shutdown user experience is now much improved - there's not much to see (just the console text from startup, which is how it should be):
This has been tested on RPi, RPi2 and Generic.
I only became aware of this shutdown issue when rebooting an i5 NUC (Generic).
Prior to using this system I hadn't seen any errors when restarting a Revo3700/ION2 (nvidia legacy) system - apparently the Revo3700 reboots before the error messages are rendered by the console. Adding a shutdown delay in
/storage/.config/shutdown.sh:revealed the shutdown errors which are identical to the NUC (see first picture, above).
With the RPi/RPi2, there are no errors visible when shutting down as the console framebuffer has already been hidden. However after analysing the journal it is evident that the same unmount
/storagefailure is occurring on RPi/RPi2.It is also apparent that attempting to unmount
/flash(throughflash.mount) on the RPi/RPi2 will also fail, since the system is still running from/flash.RPi2 shutdown log (without this PR) - edited to remove noise, showing failures to unmount
/storageand/flash:(Full log: http://sprunge.us/SfUd)
Thus
/flashis also added to the ignore list when unmounting mount points.Unmounting of
/storagealso fails in OpenELEC 6.0.3 so this isn't an entirely new phenomenon, however the shutdown sequence in OE 6.0.3 (based on systemd-219) is very different to that in LibreELEC, with the filesystem unmounts occurring later in the shutdown sequence. Although/storagecannot be unmounted in OpenELEC, it isn't being logged as a failure (for a reason I haven't been able to determine).So, as OE and LE have never actually unmounted
/storage(and also - at least on RPi/RPi2 -/flash) successfully, ignoring/storageand/flashwhen unmounting filesystems should have no noticeable effect other than to avoid failures being logged and potentially shown to the user.Ping @seo as you may have a better idea how to solve this issue...
As far as I can see, the only way to successfully unmount
/storagewould be to do so after the journal service has stopped, ie. right at the very end of the shutdown sequence. The problem with delayingumount.targetwill be that any network mounts will have difficulty unmounting as the network will already be down - this is a problem solved in OE with OpenELEC/OpenELEC.tv#4659.I also don't see how it would be possible to unmount
/flashwhen the system is still running from that mount point.Consequently reorganising the shutdown sequence to resolve these largely cosmetic issues might be more trouble than it's worth - hence this quick hack - but anything you can suggest would be welcome.
@gdachs you reported issue OpenELEC/OpenELEC.tv#4475 against OpenELEC, can you confirm if this is still an issue in LibreELEC with and without this new PR?