Fixes grub1 installer tests #46769

samueldr · 2018-09-17T04:33:14Z

Motivation for this change

For #45960, needs backport to 18.09.

This upgrades grub to the latest Debian patchset. Also tried updating to the latest Gentoo patchset, didn't work better.

Additionally, gcc6 is required for building grub1.

I'm not 100% positive on the method used to fix the installer test (creating /tmp after the partitioning), but 100% positive that it is the fix required. This was found by using grub-install --debug (through modfying the nixos grub installer script). (Note that --debug on grub-install FORCIBLY exits with error code 1.)

As for: "what broke the build?", the switch to gcc7 was the initial breakpoint. At that moment, the grub1 build was failing.

Why the segfaults aren't visible in the failure logs? I thought I had found clues in #46763, but turns out that by validating whether gcc6 was required or not, I found out that my initial assumptions were wrong. Still an open question.

Things done

✔️ Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- ✔️ NixOS
- 🆖 macOS
- ⬜ other Linux distributions
✔️ Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
⬜ Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
✔️ Tested execution of all binary files (usually in ./result/bin/)
⬜ Determined the impact on package closure size (by running nix path-info -S before and after)
✔️ Fits CONTRIBUTING.md.

samueldr · 2018-09-17T04:42:05Z

@GrahamcOfBorg test installer.grub1

GrahamcOfBorg · 2018-09-17T04:43:21Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.installer.grub1

Partial log (click to expand)

error: while evaluating anonymous function at /var/lib/gc-of-borg/nix-test-rs-16/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-16/nixos/release.nix:30:25, called from undefined position:
while evaluating 'hydraJob' at /var/lib/gc-of-borg/nix-test-rs-16/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-16/lib/customisation.nix:157:14, called from /var/lib/gc-of-borg/nix-test-rs-16/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-16/nixos/release.nix:30:28:
while evaluating the attribute 'drvPath' at /var/lib/gc-of-borg/nix-test-rs-16/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-16/lib/customisation.nix:174:13:
while evaluating the attribute 'drvPath' at /var/lib/gc-of-borg/nix-test-rs-16/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-16/lib/customisation.nix:141:13:
while evaluating the attribute 'buildCommand' of the derivation 'vm-test-run-installer-grub1' at /var/lib/gc-of-borg/nix-test-rs-16/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-16/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating the attribute 'testScript' of the derivation 'nixos-test-driver-installer-grub1' at /var/lib/gc-of-borg/nix-test-rs-16/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-16/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating 'isFunction' at /var/lib/gc-of-borg/nix-test-rs-16/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-16/lib/trivial.nix:221:16, called from /var/lib/gc-of-borg/nix-test-rs-16/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-16/nixos/lib/testing.nix:104:12:
while evaluating 'testScriptFun' at /var/lib/gc-of-borg/nix-test-rs-16/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-16/nixos/tests/installer.nix:65:19, called from /var/lib/gc-of-borg/nix-test-rs-16/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-16/nixos/tests/installer.nix:267:20:
Non-EFI boot methods are only supported on i686 / x86_64

GrahamcOfBorg · 2018-09-17T04:44:21Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: grub

Partial log (click to expand)

Cannot nix-instantiate `grub' because:
error: assertion failed at /private/var/lib/ofborg/builds/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/top-level/stage.nix:150:21

GrahamcOfBorg · 2018-09-17T04:45:10Z

Unexpected error: command failed with exit code 1 on aarch64-linux (full log)

Attempted: grub

Partial log (click to expand)

copying path '/nix/store/vg6nrfdzdkzlcya6j0s0nsa3f3la4vh2-texinfo-6.5' from 'https://cache.nixos.org'...
copying path '/nix/store/6w3mwdy2ls9akyzad2k3c499d482w5hv-openssl-1.0.2p-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/g0r4ap53f8wrggm3qz2mlcayz84lpz8x-curl-7.61.0-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/pnibzawxn2bv6c3brnvs67pbff3fhdad-binutils-wrapper-2.30' from 'https://cache.nixos.org'...
copying path '/nix/store/jx05xvfdzd0mkrc6lvwbc0qbxpmhgyma-gcc-wrapper-7.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/a5vsizg0wmk5iygz0yi0hd9g9scvk0mv-gcc-wrapper-6.4.0' from 'https://cache.nixos.org'...
copying path '/nix/store/959qszg1mwfrlvrvzjg53zaz56iyq8s3-libtool-2.4.6' from 'https://cache.nixos.org'...
copying path '/nix/store/5bd9r11dj22g75ajn4dqaz0sfb7k19g6-stdenv-linux' from 'https://cache.nixos.org'...
building '/nix/store/562f42ly42rbvi1zyxacq4zp0l2k7hhh-stdenv-linux.drv'...
error: a 'i686-linux' is required to build '/nix/store/ay1x8zf0myhpk63jh41x3fird33msiaa-patchutils-0.3.3.drv', but I am a 'aarch64-linux'

samueldr · 2018-09-17T04:49:09Z

Welp, something else on master seems wrong :/.

On top of nixos-unstable, This is the result of the tests on x86_64.

GrahamcOfBorg · 2018-09-17T04:49:54Z

Unexpected error: command failed with exit code 1 on x86_64-linux (full log)

Attempted: tests.installer.grub1

Partial log (click to expand)

100   812  100   812    0     0   6710      0 --:--:-- --:--:-- --:--:--  6655
fixed-output derivation produced path '/nix/store/cabx5xfy38637mgfyx7z7nyws522hb48-xfs_freeze.patch' with sha256 hash '0rvdy6w2yz1mkxka8kllkvwlhckf7ayx30lscy6wqi2vpm66j7gf' instead of the expected hash '1wqgj8ar4x4zwa37bj4a7kldiz5v92msigy3cv879nnk6sz4rmhg'
cannot build derivation '/nix/store/a2yhyv4k5px39xhi97sgrkgi51z45azw-grub-0.97-73.drv': 28 dependencies couldn't be built
cannot build derivation '/nix/store/vzhnqxa8br02193vin95vhz5bybzazwf-nixos-system-machine-19.03.git.3b07540.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/pyszph0bw9g4qkdl918hbi6ly0hwdhci-closure-info.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/lask30i91gswfbra8gc9h4s134r2a5j1-run-nixos-vm.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/m9klif2f5minjjyb4z3d5p33vc7mj2r8-nixos-vm.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/xrccydhknhvpic27ww82863dp8dq6hmx-nixos-test-driver-installer-grub1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/clpb7li3hpdibjvkdz73npwjp72nz41h-vm-test-run-installer-grub1.drv': 1 dependencies couldn't be built
error: build of '/nix/store/clpb7li3hpdibjvkdz73npwjp72nz41h-vm-test-run-installer-grub1.drv' failed

GrahamcOfBorg · 2018-09-17T04:50:18Z

Unexpected error: command failed with exit code 1 on x86_64-linux (full log)

Attempted: grub

Partial log (click to expand)

fixed-output derivation produced path '/nix/store/hndhd3j2pkli6l31gar7042865q9jhlr-splashimage_help.patch' with sha256 hash '042kq2480kgavnpcm3z5xmg3ph8y3xx6nxd7dl8wkawlwp8qq2pl' instead of the expected hash '1lj3xh56wf1pdcf0fg585vmggrz7qqfzbhg91qv0rf4snf3ybfvr'
building '/nix/store/802fb90m20axfpv8i3ac51cb4gzngs7j-xfs_freeze.patch.drv'...

trying https://sources.debian.org/data/main/g/grub/0.97-73/debian/patches//xfs_freeze.patch
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   812  100   812    0     0   8458      0 --:--:-- --:--:-- --:--:--  8458
fixed-output derivation produced path '/nix/store/cabx5xfy38637mgfyx7z7nyws522hb48-xfs_freeze.patch' with sha256 hash '0rvdy6w2yz1mkxka8kllkvwlhckf7ayx30lscy6wqi2vpm66j7gf' instead of the expected hash '1wqgj8ar4x4zwa37bj4a7kldiz5v92msigy3cv879nnk6sz4rmhg'
cannot build derivation '/nix/store/a2yhyv4k5px39xhi97sgrkgi51z45azw-grub-0.97-73.drv': 28 dependencies couldn't be built
error: build of '/nix/store/a2yhyv4k5px39xhi97sgrkgi51z45azw-grub-0.97-73.drv' failed

samueldr · 2018-09-17T04:58:04Z

And, apparently the source URLs for the patches isn't stable? What is going on? Checking using nix-prefetch-url and the hashes are good here? (Marked WIP because of this, going to fix this up tomorrow if they indeed are wrong...)

worldofpeace · 2018-09-17T05:21:38Z

@samueldr
I corrected the hashes for you https://gist.github.com/worldofpeace/31d9c73fdfc03cdadad0c1e95d271fc6

xeji · 2018-09-17T07:53:36Z

Thank you for debugging this! I spent some time on it and just couldn't figure out what was wrong...

xeji · 2018-09-17T07:58:23Z

pkgs/tools/misc/grub/default.nix


+let
+  baseUrl = "https://sources.debian.org/data/main/g/grub/0.97-73/debian/patches";


sources.debian.org is not version controlled, they tend to remove/change files so we get 404s and broken hashes later.

Can you please try to find the patches in their gitlab repo: https://salsa.debian.org/grub-team/grub/tree/master/debian/patches - they should all be there somewhere, although maybe not all int the grub-team project.

You can change baseUrl to -> https://salsa.debian.org/grub-team/grub-legacy/raw/debian/0.97-73/debian/patches

The corrected hashes are the same.
I also checked and all the files referenced here are there also.

Hm! Though I hadn't verified elsewhere whether it was a safe location, I verified that other locations in Nixpkgs used sources.debian.org. You say "not version controlled", but I see the version in the URL, which I would assume always match with a released version (since I see those). Is it that even though they are versioned, they may change?

We've had issues fetchpatch-ing from sources.debian.org before. The content is not guaranteed to be stable, they do sometimes change or remove files even under the same version in the URL.
It's still used in nixpkgs but let's try to avoid it in the future. So the gitlab repo is the best option if you can find the patches there.

xeji · 2018-09-17T08:00:30Z

Fixes #39025

GCC 7 wasn't kind to grub.

Instead of using Gentoo's patchset, uses Debian's. Gentoo's doesn't work anymore.

GrahamcOfBorg · 2018-09-17T13:47:09Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.installer.grub1

Partial log (click to expand)

error: while evaluating anonymous function at /var/lib/gc-of-borg/nix-test-rs-4/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-4/nixos/release.nix:30:25, called from undefined position:
while evaluating 'hydraJob' at /var/lib/gc-of-borg/nix-test-rs-4/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-4/lib/customisation.nix:157:14, called from /var/lib/gc-of-borg/nix-test-rs-4/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-4/nixos/release.nix:30:28:
while evaluating the attribute 'drvPath' at /var/lib/gc-of-borg/nix-test-rs-4/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-4/lib/customisation.nix:174:13:
while evaluating the attribute 'drvPath' at /var/lib/gc-of-borg/nix-test-rs-4/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-4/lib/customisation.nix:141:13:
while evaluating the attribute 'buildCommand' of the derivation 'vm-test-run-installer-grub1' at /var/lib/gc-of-borg/nix-test-rs-4/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-4/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating the attribute 'testScript' of the derivation 'nixos-test-driver-installer-grub1' at /var/lib/gc-of-borg/nix-test-rs-4/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-4/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating 'isFunction' at /var/lib/gc-of-borg/nix-test-rs-4/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-4/lib/trivial.nix:221:16, called from /var/lib/gc-of-borg/nix-test-rs-4/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-4/nixos/lib/testing.nix:104:12:
while evaluating 'testScriptFun' at /var/lib/gc-of-borg/nix-test-rs-4/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-4/nixos/tests/installer.nix:65:19, called from /var/lib/gc-of-borg/nix-test-rs-4/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-4/nixos/tests/installer.nix:267:20:
Non-EFI boot methods are only supported on i686 / x86_64

GrahamcOfBorg · 2018-09-17T14:00:18Z

Success on x86_64-linux (full log)

Attempted: tests.installer.grub1

Partial log (click to expand)

machine# [   14.813833] sd 0:0:0:0: [sda] Synchronizing SCSI cache
machine# [   14.814766] sd 0:0:0:0: [sda] Stopping disk
machine# [   14.815788] ACPI: Preparing to enter system sleep state S5
machine# [   14.816610] reboot: Power down
collecting coverage data
syncing
test script finished in 418.41s
vde_switch: EOF on stdin, cleaning up and exiting
cleaning up
/nix/store/3xswvllgxqss6k3ccp66xj4428fkamy9-vm-test-run-installer-grub1

samueldr · 2018-09-17T14:10:51Z

Ah, as far as the hashes are concerned: it's a difference I didn't know about fetchpatch and fetchurl. fetchpatch will normalize the patches, as documented, and the sha256 is the normalized patch. As far as those files are concerned, fetchurl should be the better alternative: they are already files and not generated data from source control (which fetchpatch seems to be built for).

I have also included the script I used to generate the file, and made it work more like the existing maintainers script; but made specially for this use. It might be wise to check into refactoring the maintainers script so it works using salsa.debian.org instead of the (apparently) problematic sources.debian.org url.

xeji · 2018-09-17T17:10:59Z

I also ran nixos.tests.installer.grub1.i686-linux locally 👍

xeji · 2018-09-17T17:14:43Z

backported in 409abea..95b11f3

samueldr mentioned this pull request Sep 17, 2018

Installer tests don't show segfaults since the transition to nix 2.0 and nixos-enter #46763

Closed

GrahamcOfBorg added the 6.topic: nixos label Sep 17, 2018

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Sep 17, 2018

samueldr changed the title ~~Fixes grub1 installer tests~~ [WIP] Fixes grub1 installer tests Sep 17, 2018

samueldr force-pushed the zhf/grub branch from a9dd3a0 to b35f2b7 Compare September 17, 2018 04:52

xeji reviewed Sep 17, 2018

View reviewed changes

samueldr force-pushed the zhf/grub branch from b35f2b7 to 1774ceb Compare September 17, 2018 12:48

samueldr added 3 commits September 17, 2018 09:31

grub: Use gcc6

ba09b05

GCC 7 wasn't kind to grub.

grub: grub-0.97-patch-1.15 -> grub-0.97-73

932c8f4

Instead of using Gentoo's patchset, uses Debian's. Gentoo's doesn't work anymore.

nixos/tests/installer: grub1 needs /tmp to exist.

b63c539

samueldr force-pushed the zhf/grub branch from 1774ceb to b63c539 Compare September 17, 2018 13:32

samueldr changed the title ~~[WIP] Fixes grub1 installer tests~~ Fixes grub1 installer tests Sep 17, 2018

xeji approved these changes Sep 17, 2018

View reviewed changes

xeji merged commit 8a2bf13 into NixOS:master Sep 17, 2018

This was referenced Sep 17, 2018

nixos-install fails with grub1 #39025

Closed

Fix failing NixOS tests #40257

Closed

samueldr deleted the zhf/grub branch February 12, 2019 02:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes grub1 installer tests #46769

Fixes grub1 installer tests #46769

samueldr commented Sep 17, 2018 •

edited

samueldr commented Sep 17, 2018 •

edited

GrahamcOfBorg commented Sep 17, 2018

GrahamcOfBorg commented Sep 17, 2018

GrahamcOfBorg commented Sep 17, 2018

samueldr commented Sep 17, 2018

GrahamcOfBorg commented Sep 17, 2018

GrahamcOfBorg commented Sep 17, 2018

samueldr commented Sep 17, 2018

worldofpeace commented Sep 17, 2018

xeji commented Sep 17, 2018

xeji Sep 17, 2018

worldofpeace Sep 17, 2018 •

edited

samueldr Sep 17, 2018

xeji Sep 17, 2018

xeji commented Sep 17, 2018

GrahamcOfBorg commented Sep 17, 2018

GrahamcOfBorg commented Sep 17, 2018

samueldr commented Sep 17, 2018

xeji commented Sep 17, 2018

xeji commented Sep 17, 2018


		let
		baseUrl = "https://sources.debian.org/data/main/g/grub/0.97-73/debian/patches";

Fixes grub1 installer tests #46769

Fixes grub1 installer tests #46769

Conversation

samueldr commented Sep 17, 2018 • edited

Motivation for this change

Things done

samueldr commented Sep 17, 2018 • edited

GrahamcOfBorg commented Sep 17, 2018

GrahamcOfBorg commented Sep 17, 2018

GrahamcOfBorg commented Sep 17, 2018

samueldr commented Sep 17, 2018

GrahamcOfBorg commented Sep 17, 2018

GrahamcOfBorg commented Sep 17, 2018

samueldr commented Sep 17, 2018

worldofpeace commented Sep 17, 2018

xeji commented Sep 17, 2018

xeji Sep 17, 2018

Choose a reason for hiding this comment

worldofpeace Sep 17, 2018 • edited

Choose a reason for hiding this comment

samueldr Sep 17, 2018

Choose a reason for hiding this comment

xeji Sep 17, 2018

Choose a reason for hiding this comment

xeji commented Sep 17, 2018

GrahamcOfBorg commented Sep 17, 2018

GrahamcOfBorg commented Sep 17, 2018

samueldr commented Sep 17, 2018

xeji commented Sep 17, 2018

xeji commented Sep 17, 2018

samueldr commented Sep 17, 2018 •

edited

samueldr commented Sep 17, 2018 •

edited

worldofpeace Sep 17, 2018 •

edited