Cannot mount xfs root partition with kernel 4.19 or later

[wirthan]

Issue description

After upgrading from 18.09 to 19.03 my system fails to mount the root partition (xfs).

It runs fsck, and later fails with:

mount: mounting /dev/disk/by-uuid/??? on /mnt-root/ failed: Invalid argument
An error occured in stage 1 of the boot process ...

Steps to reproduce

nix-channel --add https://nixos.org/channels/nixos-19.03 nixos
nix-channel --update
nixos-rebuild switch

Technical details

hardware-configuration.nix:

boot.kernelModules = [ "kvm-intel" "xfs" ];
fileSystems."/" =
  { device = "/dev/disk/by-uuid/32efabe2-911b-4da9-a9e1-13ec93dc1b4d";
    fsType = "xfs";
    options = ["defaults" "discard" "noatime" "nobarrier"];
  };

/nix/store/rlfna41bld4wlbxvg3yjackkl3nssl7w-etc-fstab:

/dev/disk/by-uuid/32efabe2-911b-4da9-a9e1-13ec93dc1b4d / xfs defaults,discard,noatime,nobarrier 0 1
/dev/disk/by-uuid/D9A3-1F95 /boot vfat defaults 0 2
/dev/disk/by-uuid/b7490d95-2dc0-4b5b-b1ae-b5d0690467b0 /home xfs defaults,discard,noatime,nobarrier 0 2
/dev/disk/by-uuid/1028883d-0360-4779-a5ea-ee2950d6f415 /var/lib/containers xfs defaults,discard,noatime,nobarrier 0 2
/dev/disk/by-label/swap none swap

[emmanuelrosa]

Do the UUID's in your configuration correspond to the output of lsblk -f? I don't see why they would change, but it's worth confirming.

[wirthan]

Yes. Actually, I mount the partitions by-label, but changed temporary to UUID in order to figure out which one of the generated fstab files is used. The issue originally appeared with the by-label setup.

[emmanuelrosa]

Do the filesystems also fail to mount when attempting to mount with UUIDs?

[wirthan]

Thanks for your answer. I can only try with 18.09. No problems here mounting the root partition by label or UUID.
Because I could not spot the "noatime" option in the kernel xfs docs, I also tried removing it - did not change anything. Havn't tried removing the "defaults" option though. The other ones are mentioned in the docs.

[emmanuelrosa]

Unfortunately it's going to be rather challenging to troubleshoot the issue if you cannot reproduce it. You may have to do some digging to find additional information we can work with.

[wirthan]

@emmanuelrosa, I think you misunderstood. The issue is perfectly reproducible: As soon as I switch to 19.03 my system will not boot anymore. Because I simply cannot try to mount a partition on a system that is not bootable, I tested the correctness of the device files by using 18.09. And that works perfectly.

But actually I made some progress: By switching back to kernel 4.14, my system can boot.
Maybe I should change the title to "Cannot mount xfs root partition with kernel 4.19".

[wirthan]

The issue still persists and is also observable with kernel 5.1.8 installed using boot.kernelPackages = pkgs.linuxPackages_latest .

Any ideas how to investigate that?

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[Atemu]

Boot with boot.shell_on_fail and check what's up in dmesg. I doubt this is a NixOS issue though.

If you have more concrete info and ideally a way we could reproduce this in a VM, please re-open the issue.