NixOS: nofail option breaks remounting changed filesystems

[bts]

Issue description

When a fileSystems entry is changed and the "nofail" option is specified, switching to the new configuration fails because mount doesn't recognize the "nofail" option (though /etc/fstab does.)

The "nofail" option should only be used to populate /etc/fstab, and not provided to mount(8) when remounting.

Steps to reproduce

1. Add a fileSystems entry using "nofail" to /etc/nixos/configuration.nix like:

{
  fileSystems."/mnt/mac-home" =
    { fsType = "vboxsf";
      device = "mac-home";
      options = [ "ro" "uid=1001" "gid=100" "nofail" ];
    };
}

2. sudo nixos-rebuild switch

3. Change the fileSystems entry (e.g. remove the uid=1001 entry above)

4. sudo nixos-rebuild switch

At this point we get an error like the following:

● mnt-mac\x2dhome.mount - /mnt/mac-home
   Loaded: loaded (/etc/fstab; generated; vendor preset: enabled)
   Active: active (mounted) (Result: exit-code) since Sat 2017-11-04 16:07:59 EDT; 1min 13s ago
    Where: /mnt/mac-home
     What: mac-home
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)
  Process: 5834 ExecRemount=/nix/store/baixlwb364lh3inyy4pcgdfacmg3amib-util-linux-2.30-bin/bin/mount mac-home /mnt/mac-home -o remount,ro,nofail -t vboxsf (code=exited, status=1/FAILURE)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/mnt-mac\x2dhome.mount

Nov 04 16:07:59 vbox systemd[1]: Mounting /mnt/mac-home...
Nov 04 16:07:59 vbox systemd[1]: Mounted /mnt/mac-home.
Nov 04 16:08:43 vbox systemd[1]: Reloading /mnt/mac-home.
Nov 04 16:08:43 vbox systemd[1]: mnt-mac\x2dhome.mount: Mount process exited, code=exited status=1
Nov 04 16:08:43 vbox systemd[1]: Reload failed for /mnt/mac-home.

Running this command (mount mac-home /mnt/mac-home -o remount,ro,nofail -t vboxsf) by hand produces the following output:

unknown mount option `nofail'
valid options:
  rw         mount read write (default)
  ro         mount read only
  uid       =<arg> default file owner user id
  gid       =<arg> default file owner group id
  ttl       =<arg> time to live for dentry
  iocharset =<arg> i/o charset (default utf8)
  convertcp =<arg> convert share name from given charset to utf8
  dmode     =<arg> mode of all directories
  fmode     =<arg> mode of all regular files
  umask     =<arg> umask of directories and regular files
  dmask     =<arg> umask of directories
  fmask     =<arg> umask of regular files

I am able to successfully switch to the new configuration if I first manually unmount the partition (umount /mnt/mac-home).

Technical details

• System: NixOS 17.09.1756.c99239bca0 (Hummingbird)
• Nix version: nix-env (Nix) 1.11.15
• Nixpkgs version: 17.09.1756.c99239bca0
• Sandboxing enabled: false

[grische]

@bts I am unable to reproduce the above issue with current NixOS.

Technical details

• system: "x86_64-linux"
• host os: Linux 4.14.87, NixOS, 18.09.git.d56ec49 (Jellyfish)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.1.3
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[Luflosi]

I can reproduce the problem. It is caused by the mount.vboxsf command not supporting the nofail option. The fstab is converted to systemd.mount units by the systemd-fstab-generator, which then call the specific mount command, leading to the failure. At least some fuse filesystems (tested with apfs-fuse) have a similar problem. I think the best solution would be to convince the developers of these mount commands to implement support for the nofail option or at least ignore this option as the man page for the mount command says, that the nofail option is supported by every filesystem.

[stale]

I marked this as stale due to inactivity. → More info

[chasecaleb]

Not stale. This also occurs for VMware guests due to unsupported remount option. Here's a trivial reproduction if you have virtualisation.vmware.guest = true:

$ sudo systemctl reload run-vmblock\\x2dfuse.mount; journalctl -u run-vmblock\\x2dfuse.mount --since "-5s"
Job failed. See "journalctl -xe" for details.
Dec 20 15:02:43 my-name systemd[1]: Reloading VMware vmblock fuse mount...
Dec 20 15:02:43 my-name mount[101993]: fuse: unknown option(s): `-o remount'
Dec 20 15:02:43 my-name systemd[1]: run-vmblock\x2dfuse.mount: Mount process exited, code=exited, status=3/NOTIMPLEMENTED
Dec 20 15:02:43 my-name systemd[1]: Reload failed for VMware vmblock fuse mount.

[chasecaleb]

Here's a stupid-but-simple workaround: if the mount unit changed, preemptively stop it in an activation script. This makes NixOS start the unit instead of attempting and failing to reload it.

Here's an example for VMware guest's run-vmblock\\x2dfuse.mount unit. Adjust the name variable as needed if you're using this for a different unit:

system.activationScripts.vmware-mount-reload-workaround = {
  deps = [ "etc" ];
  text = ''
    name=run-vmblock\\x2dfuse.mount
    path=etc/systemd/system/$name
    old_mount=/run/current-system/$path
    new_mount=$systemConfig/$path

    if ! ${pkgs.diffutils}/bin/diff "$old_mount" "$new_mount" >/dev/null 2>&1; then
      ${pkgs.systemd}/bin/systemctl stop "$name"
    fi
  '';
};

[chasecaleb]

I reported a bug regarding remount to fuse since it appears to be their issue: libfuse/libfuse#717

[gunar]

For me the nofail option isn't observed at all. The machine fails to boot up if I remove the USB SSD.

This is my workaround:

  # XXX: This isn't working
  # fileSystems."/backup" = {
  #   device = "/dev/mapper/ssd";
  #   options = [ "nofail" ];
  # };
 systemd.services.myBackupMount = {
    description = "Mount /backup";
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = "yes";
      ExecStart = "${pkgs.utillinux}/bin/mount /dev/mapper/ssd /backup";
      ExecStop = "${pkgs.utillinux}/bin/umount /backup";
    };
    unitConfig = {
      ConditionPathExists = "/dev/mapper/ssd";
    };
  };

[nioncode]

For me the nofail option isn't observed at all. The machine fails to boot up if I remove the USB SSD.

I just set up a fresh system and have multiple optional disks configured with nofail and this is respected during boot and shows up in journalctl as:

Dez 29 00:39:16 servnix stage-1-init: [Thu Dec 28 23:39:08 UTC 2023] Device /dev/disk/by-uuid/334f416d-cdb6-493f-adbc-3ff33e688690 does not exist >
Dez 29 00:39:16 servnix stage-1-init: [Thu Dec 28 23:39:08 UTC 2023] Device /dev/disk/by-uuid/498cbd1a-9606-40b4-9180-9581b9c34340 does not exist >
Dez 29 00:39:16 servnix stage-1-init: [Thu Dec 28 23:39:08 UTC 2023] Device /dev/disk/by-uuid/8288607c-34aa-45de-8631-5cabc8d58b61 does not exist >

HOWEVER, when switching to a changed configuration, this (at least initially) fails because it tries to start the mount units of the non-existing drives, which fails:

Failed to open /nix/store/5rrvr6b780h8lz4msgy7wpm1sarh0q5b-nixos-system-servnix-23.11.2217.d02d818f22c7/etc/systemd/system/mnt-media-4tb.mount: No such file or directory at /nix/store/5rrvr6b780h8lz4msgy7wpm1sarh0q5b-nixos-system-servnix-23.11.2217.d02d818f22c7/bin/switch-to-configuration line 219.
starting the following units: mnt-media-4tb.mount
A dependency job for mnt-media-4tb.mount failed. See 'journalctl -xe' for details.
warning: error(s) occurred while switching to the new configuration

I cannot reproduce this issue though, so it might have something to do with the initial switch or so? After a reboot, switching to new configurations works fine.