dracut-056-1 an issue with systemd.volatile=overlay, nfsroot and systemd-volatile-root.service

[0n0w1c]

Describe the bug
system-volatile-root.service requires /sysroot to be mounted

Distribution used
Manjaro

Dracut version
056-1

Init system
systemd

To Reproduce
Add systemd.volatile=overlay to kernel command line of a functional nfsroot setup:
root=10.1.1.2:/srv/nfs/root systemd.volatile=overlay

Expected behavior
An overlayfs on top of the nfsroot

Additional context
I added a small shell script to bring up the network and mount the nfsroot on /sysroot as a ExecStartPre in systemd-volatile-root.service and this proved to be successful. I had a ticket open with systemd thinking this was a systemd issue, but was informed it is a dracut issue, as dracut is responsible for the nfsroot mount.

systemd.volatile=overlay works fine without modification for a local root filesystem, the issue is only with an NFS root.

[johannbg]

It would be nice if you could actually provide the script that you use with systemd type unit, the dracut conf and other relevant part to your setup and logs.

That said I suspect this which checks for the existence of $NEWROOT/proc under the assumption that the new root filesystem is already mounted so test commenting that line out and see if that works with systemd.volatile=overlay

#echo '[ -e $NEWROOT/proc ]' > $hookdir/initqueue/finished/nfsroot.sh

[0n0w1c]

Thank you for the reply, I will attempt your suggestion shortly and report back.

I hardcoded this simple ExecStartPre script and placed it in systemd-volatile-root.service:

/usr/lib/systemd/system/pre-mount

#!/bin/sh 

# use eth0, not yet renamed to enp1s0
ifup eth0

ip route add default via 10.1.1.1

rpcbind

rpc.statd

mount 10.1.1.2:/srv/nfs/root /sysroot

More details can be found here, it is the ticket I had opened with systemd.

[0n0w1c]

Unfortunately, commenting out that particular line did not work for my setup, I received the same error:

systemd[1]: Failed to start Enforce Volatile Root File Systems.

In journalctl -b I find:

systemd-volatile-root[207]: /sysroot is not a mount point.

This error occurs very early in the process, prior to the 95nfs hook. If I place a rd.break=initqueue, it does enter maintenance mode, just before the error is reached.

[0n0w1c]

I have not previously noted how I build the initrd:

dracut --no-hostonly --no-hostonly-cmdline --no-early-microcode /boot/initramfs-dracut.img 5.15.26-1-MANJARO

I also needed to add sm-notify to the initrd, but that may just be an issue with dracut on Manjaro:

/etc/dracut.conf.d/overlay.conf

add_dracutmodules+=" network-legacy "
install_items+=" /usr/bin/sm-notify /usr/lib/systemd/system/pre-mount "

[0n0w1c]

In my script, ifup eth0 brings up the network and works fine with dhcp for the IP address, however the gateway (default route) is not assigned. While I will modify my script to parse the IP and generate the gateway (all of ours are .1), this is not generally useful. I am hopeful a proper solution will be devised.

/bin/ifup being the shell script dracut places in the initrd, not the ifup/ifdown of Ubuntu.

[0n0w1c]

After a little more testing, it seems systemd-volatile-root.service is visited twice in the boot process... the second time is when the normal nfs root mount occurs. Normal meaning without the overlay. And it needed little cleanup and a bit of error checking, so I reworked my script a bit, to exit if /sysroot is already mounted.

#!/bin/sh

mount | grep sysroot > /dev/null
if [ $? -eq 0 ]; then
   exit 0
fi

IFACE=eth0
ifup ${IFACE}
if [ $? -ne 0 ]; then
   exit 1
fi

GATEWAY=$(ip address | grep ${IFACE} | grep inet | awk -F "/" '{ print $1 }' | awk -F "inet " '{ print $2 }' | awk -F "." '{ print $1"."$2"."$3 }').1
ip route add default via ${GATEWAY}
if [ $? -ne 0 ]; then
   exit 1
fi

NFS_MOUNT=$(cat /proc/cmdline | awk -F "root=" '{ print $2 }' | awk -F " " '{ print $1 }')
mount ${NFS_MOUNT} /sysroot
if [ $? -ne 0 ]; then
   exit 1
fi

[0n0w1c]

Any traction on this issue or should I plan to use my hack-around?

[LaszloGombos]

@0n0w1c - is this by any chance related - #1759 ? Are you in a position to help to test ?