Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Impurity causing initrd failing to build (due to filesystem/ext4) #15581

Closed
domenkozar opened this issue May 20, 2016 · 6 comments
Closed

Impurity causing initrd failing to build (due to filesystem/ext4) #15581

domenkozar opened this issue May 20, 2016 · 6 comments

Comments

@domenkozar
Copy link
Member

@domenkozar domenkozar commented May 20, 2016

Two different machines, same nixpkgs commit 6a142de (release-16.03)

On failing machine:

$ NIXOS_CONFIG=`pwd`/nixos/modules/virtualisation/nova-image.nix nix-build nixos -A config.system.build.initialRamdisk
these derivations will be built:
  /nix/store/szsgrz95ri6b8nafb2hjx4fysjzm8w6d-initrd.drv
building path(s) ‘/nix/store/6lcvva6gps6sy7kfy8z3r5z77gc1077l-initrd’
cp: missing destination file operand after '.'
Try 'cp --help' for more information.
builder for ‘/nix/store/szsgrz95ri6b8nafb2hjx4fysjzm8w6d-initrd.drv’ failed with exit code 1
error: build of ‘/nix/store/szsgrz95ri6b8nafb2hjx4fysjzm8w6d-initrd.drv’ failed

On working machine:

$ NIXOS_CONFIG=`pwd`/nixos/modules/virtualisation/nova-image.nix nix-build nixos -A config.system.build.initialRamdisk -K --check
checking path(s) ‘/nix/store/6lcvva6gps6sy7kfy8z3r5z77gc1077l-initrd’
30648 blocks
note: keeping build directory ‘/tmp/nix-build-initrd.drv-2’
error: derivation ‘/nix/store/szsgrz95ri6b8nafb2hjx4fysjzm8w6d-initrd.drv’ may not be deterministic: output ‘/nix/store/6lcvva6gps6sy7kfy8z3r5z77gc1077l-initrd’ differs from ‘/nix/store/6lcvva6gps6sy7kfy8z3r5z77gc1077l-initrd-check’

Observations

  • both using chroot
  • both have the same hash

The failure seems to happen due to following line returning empty string in file pkgs/build-support/kernel/make-initrd.sh:

storePaths=$(perl $pathsFromGraph closure-*)

I'll check the difference in $sourceRoot now, but the conents should be the same?

@domenkozar
Copy link
Member Author

@domenkozar domenkozar commented May 20, 2016

It's ext4 and it's corrupted files (usually happens on sudden power loss).

nix-store --repair-path /nix/store/igp3zlizdgjr5zg041c5a5rnp24pimmb-paths-from-graph.pl fixes the issue

@dezgeg
Copy link
Contributor

@dezgeg dezgeg commented May 20, 2016

Well, I think that is indirectly caused by Nix not doing enough fsync() calls. That is, it's not an ext4 bug but rather working as expected (from their point of view).

@domenkozar
Copy link
Member Author

@domenkozar domenkozar commented May 20, 2016

@dezgeg I've had this a few times, if you can fix it I'd be grateful forever :)

@domenkozar
Copy link
Member Author

@domenkozar domenkozar commented May 22, 2016

@dezgeg very relevant: https://www.usenix.org/node/186195 (All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications)

@domenkozar domenkozar changed the title Impurity causing initrd failing to build. Impurity causing initrd failing to build (due to filesystem/ext4) May 22, 2016
@dezgeg
Copy link
Contributor

@dezgeg dezgeg commented May 22, 2016

Yes I know it's tricky business in the general case, but AFAICT for Nix the only thing needed is a depth-first fsync of the output path before registering the paths in the SQLite database. Seems like there's even currently an undocumented option sync-before-registering to use the big hammer and call sync(), but that's likely not very performant. What dpkg seems to do in addition to fsync is to use sync_file_range to improve performance: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=605009

domenkozar added a commit to domenkozar/nixpkgs that referenced this issue May 23, 2016
The motivation is using sudo in chroot nix builds, a somewhat
special edge case I have and pulling system path into chroot
yields to some very nasty bug like
NixOS#15581

Previously:

$ cat /var/setuid-wrappers/sudo.real
/nix/store/3sm04dzh0994r86xqxy52jjc0lqnkn65-system-path/bin/sudo

After the change:

$ cat /var/setuid-wrappers/sudo.real
/nix/store/4g9sxbzy8maxf1v217ikp69c0c3q12as-sudo-1.8.15/bin/sudo
@domenkozar
Copy link
Member Author

@domenkozar domenkozar commented May 23, 2016

It turns out the issue was completely different impurity: snabblab/snabblab-nixos#42 (comment)

I'll open a trivial PR soon to address this (testing now if it works)

domenkozar added a commit that referenced this issue May 23, 2016
The motivation is using sudo in chroot nix builds, a somewhat
special edge case I have and pulling system path into chroot
yields to some very nasty bug like
#15581

Previously:

$ cat /var/setuid-wrappers/sudo.real
/nix/store/3sm04dzh0994r86xqxy52jjc0lqnkn65-system-path/bin/sudo

After the change:

$ cat /var/setuid-wrappers/sudo.real
/nix/store/4g9sxbzy8maxf1v217ikp69c0c3q12as-sudo-1.8.15/bin/sudo
domenkozar added a commit that referenced this issue May 23, 2016
The motivation is using sudo in chroot nix builds, a somewhat
special edge case I have and pulling system path into chroot
yields to some very nasty bug like
#15581

Previously:

$ cat /var/setuid-wrappers/sudo.real
/nix/store/3sm04dzh0994r86xqxy52jjc0lqnkn65-system-path/bin/sudo

After the change:

$ cat /var/setuid-wrappers/sudo.real
/nix/store/4g9sxbzy8maxf1v217ikp69c0c3q12as-sudo-1.8.15/bin/sudo
wkennington pushed a commit to triton/triton that referenced this issue May 25, 2016
The motivation is using sudo in chroot nix builds, a somewhat
special edge case I have and pulling system path into chroot
yields to some very nasty bug like
NixOS/nixpkgs#15581

Previously:

$ cat /var/setuid-wrappers/sudo.real
/nix/store/3sm04dzh0994r86xqxy52jjc0lqnkn65-system-path/bin/sudo

After the change:

$ cat /var/setuid-wrappers/sudo.real
/nix/store/4g9sxbzy8maxf1v217ikp69c0c3q12as-sudo-1.8.15/bin/sudo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants