transient failure in nixos installer tests: swap.target unit fails to start

[domenkozar]

Example: http://hydra.nixos.org/build/17640976/log/tail-reload

cc @wkennington

[wkennington]

It actually has nothing to do with btrfs, and all to do with the swap startup unit failing. This happens for all of the installer tests somewhat randomly. I have absolutely no idea why swap would sometimes not come up at boot. From what I can tell, it looks like it is trying to mount the swap partition before the device is available.

[domenkozar]

Yup.

machine# systemd[1]: Found device /dev/vda2.
machine# [    4.009341] systemd[1]: Found device /dev/ttyS0.
machine# [    4.036076] systemd[1]: Unit dev-vda2.swap entered failed state.
machine# [    4.042395] Adding 1048572k swap on /dev/vda2.  Priority:-1 extents:1 across:1048572k 
machine# [    4.050677] ip6_tables: (C) 2000-2006 Netfilter Core Team
machine# [    4.040329] systemd[1]: Found device /dev/disk/by-uuid/b0215697-759e-49d3-b722-ae33a4276b5f.
machine# [    4.061244] systemd[1]: Found device /dev/hvc0.
machine# [    4.073185] systemd[1]: Mounting /boot...
machine# [    4.089654] systemd[1]: Starting backdoor.service...
machine# [    4.101399] systemd[1]: Started backdoor.service.
machine# [    4.112665] systemd[1]: Activating swap Swap Partition...
machine# [    4.124547] systemd[1]: Activating swap /dev/disk/by-uuid/165fe56e-11b4-4da9-83d9-6ac54215d75b...
machine# [    4.135083] systemd[1]: dev-vda2.swap swap process exited, code=exited status=255
machine# [    4.147351] systemd[1]: Failed to activate swap Swap Partition.

[wkennington]

This is caused by an ordering issues with the systemd autogenerate swap mountpoints. If the systemd autogenerate swap mount preceeds the fstab swap entry at boot it will cause the fstab entry to fail which breaks booting of the system and causes this transient failure.

My current idea for a workaround is to simply disable autogeneration of swap devices by nixos-generate-config and let the automounter in systemd do its thing. However, this doesn't fix the case where a user specifies a device and it gets preempted by systemd.

There should be a way to enforce a specific ordering where all of the user defined swap mountpoints come before systemd tries to automount swaps.

[lucabrunox]

Ordering can be achieved by writing systemd units instead of fstab entries. If we want to keep fstab entries, a possibility is to force the override of units when specifying systemd units in the nixos configuration. That way, fstab entry remains as is, and the systemd override only specifies orderings.

[wkennington]

The problem is, there is no specific service / mount which you can order against since the generator is called directly by init and the mount definitions are autogenerated at runtime.

[lucabrunox]

@wkennington you can override in /etc/systemd/system/foo.swap.d/ and tell the orders in [Unit]

[wkennington]

@lethalman This is actually a bug in systemd so I'm going to work with upstream to fix it.