qubes-mount-dir not starting in R3.2 - noauto absent from fstab

[ptitdoc]

Qubes OS version (e.g., R3.2):

R3.2

Affected TemplateVMs (e.g., fedora-23, if applicable):

archlinux

---

Expected behavior:

Template boot properly with last qubes-agent

Actual behavior:

Template boots but qubes-mount-dirs does not start. Because of that /home/ is empty and gui-agent fails

Steps to reproduce the behavior:

This problem occurs sometimes, without any identified reason. This may be a race condition. This has also been observed by @cryptix in issue #2609.

General notes:

# qvm-run mybrokenvm --pass-io 'sudo journalctl -la -u qubes-mount-dirs'
[...]
Jan 30 14:14:39 localhost mount-dirs.sh[353]: Private device size management: enlarging /dev/xvdb
Jan 30 14:14:39 localhost systemd[1]: Starting Initialize and mount /rw and /home...
Jan 30 14:14:39 localhost mount-dirs.sh[353]: Private device size management: resize2fs /dev/xvdb failed:
Jan 30 14:14:39 localhost mount-dirs.sh[353]: resize2fs 1.43.3 (04-Sep-2016)
Jan 30 14:14:39 localhost mount-dirs.sh[353]: resize2fs: Device or resource busy while trying to open /dev/xvdb
Jan 30 14:14:39 localhost mount-dirs.sh[353]: Couldn't find valid filesystem superblock.
Jan 30 14:14:39 localhost mount-dirs.sh[353]: Private device size management: attempting to mark /dev/xvdb clean
Jan 30 14:14:39 localhost mount-dirs.sh[353]: Private device size management: /dev/xvdb could not be marked clean:
Jan 30 14:14:39 localhost mount-dirs.sh[353]: /dev/xvdb is in use.
Jan 30 14:14:39 localhost mount-dirs.sh[353]: e2fsck: Cannot continue, aborting.
Jan 30 14:14:39 localhost mount-dirs.sh[353]: Private device size management: expect serious trouble ahead
Jan 30 14:14:39 localhost mount-dirs.sh[353]: mount: /dev/xvdb is already mounted or /rw busy
Jan 30 14:14:39 localhost systemd[1]: qubes-mount-dirs.service: Main process exited, code=exited, status=32/n/a
Jan 30 14:14:39 localhost systemd[1]: Failed to start Initialize and mount /rw and /home.
Jan 30 14:14:39 localhost systemd[1]: qubes-mount-dirs.service: Unit entered failed state.
Jan 30 14:14:39 localhost systemd[1]: qubes-mount-dirs.service: Failed with result 'exit-code'.

---

Related issues:

#2609

[marmarek]

/usr/lib/qubes/init/mount-dirs.sh is responsible for mounting /rw. I guess in Archlinux some other entity also mount it. Is "noauto" option missing in /etc/fstab.

[marmarek]

/usr/lib/qubes/init/mount-dirs.sh is responsible for mounting /rw. I guess in Archlinux some other entity also mount it. Is "noauto" option missing in /etc/fstab.

[ptitdoc]

No, noauto is not enabled for /rw. I'm trying to enable it.

Probably sometimes archlinux manages to mount it before through fstab generator. Maybe we can enforce the qubes-mount-dir service to be started before the fstab generator mount target ?

$ qvm-run --pass-io mybrokenvm 'sudo cat /etc/fstab'
su: warning: cannot change directory to /home/user: No such file or directory
# 
# /etc/fstab: static file system information
#
# <file system>		<dir>		<type>	<options>		<dump>	<pass>
# Qubes Agents
xen					/proc/xen		xenfs	defaults				0 0

# Template Directories
/dev/mapper/dmroot	/			ext4		defaults,noatime		1 1
/dev/xvdb				/rw			ext4		defaults,noatime		1 2
/dev/xvdc1			swap		swap	defaults				0 0

# Template Binds
/rw/home				/home		none	noauto,bind,defaults	0 0

# Custom settings
tmpfs				/tmp			tmpfs	nodev,nosuid,size=1G	0 0

[ptitdoc]

No, noauto is not enabled for /rw. I'm trying to enable it.

Probably sometimes archlinux manages to mount it before through fstab generator. Maybe we can enforce the qubes-mount-dir service to be started before the fstab generator mount target ?

$ qvm-run --pass-io mybrokenvm 'sudo cat /etc/fstab'
su: warning: cannot change directory to /home/user: No such file or directory
# 
# /etc/fstab: static file system information
#
# <file system>		<dir>		<type>	<options>		<dump>	<pass>
# Qubes Agents
xen					/proc/xen		xenfs	defaults				0 0

# Template Directories
/dev/mapper/dmroot	/			ext4		defaults,noatime		1 1
/dev/xvdb				/rw			ext4		defaults,noatime		1 2
/dev/xvdc1			swap		swap	defaults				0 0

# Template Binds
/rw/home				/home		none	noauto,bind,defaults	0 0

# Custom settings
tmpfs				/tmp			tmpfs	nodev,nosuid,size=1G	0 0

[marmarek]

See here for reference fstab: https://github.com/QubesOS/qubes-core-agent-linux/blob/master/misc/fstab

[marmarek]

See here for reference fstab: https://github.com/QubesOS/qubes-core-agent-linux/blob/master/misc/fstab

[ptitdoc]

This should be at least fixed in new templates.

I however like to see it fixed through systemd dependencies. I will test such a fix if I found the time.

[ptitdoc]

This should be at least fixed in new templates.

I however like to see it fixed through systemd dependencies. I will test such a fix if I found the time.

[ptitdoc]

The strange thing is that a dependency has been properly added for rw.mount and home.mount. This dependency is apparently not enforced by systemd:

https://github.com/QubesOS/qubes-core-agent-linux/blob/master/vm-systemd/qubes-mount-dirs.service

[ptitdoc]

The strange thing is that a dependency has been properly added for rw.mount and home.mount. This dependency is apparently not enforced by systemd:

https://github.com/QubesOS/qubes-core-agent-linux/blob/master/vm-systemd/qubes-mount-dirs.service

[ptitdoc]

Maybe we have to reference rw.automount home.automount instead ? I'm currently not able to reproduce the bug, so I cannot verify this fact.
One option would also be adding Conflicts with home.automount and rw.automount.

[ptitdoc]

Maybe we have to reference rw.automount home.automount instead ? I'm currently not able to reproduce the bug, so I cannot verify this fact.
One option would also be adding Conflicts with home.automount and rw.automount.