Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

linuxHeaders: 4.19.16 -> 5.5 #78994

Merged
merged 2 commits into from Feb 11, 2020
Merged

linuxHeaders: 4.19.16 -> 5.5 #78994

merged 2 commits into from Feb 11, 2020

Conversation

@orivej
Copy link
Contributor

@orivej orivej commented Feb 1, 2020

Linux has reworked its header install process in
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=59b2bd05f5f4dc62979c2e82ddd384f07e8f10bc
It has introduced make headers that prepares the headers in ./usr/include, and
made make headers_install copy the headers from there using rsync.

Motivation for this change

This is needed to use bpf/xsk.h from libbpf 0.6: libbpf/libbpf#74 (comment) (unless we install replacement linux headers with libbpf).

This obsoletes #77103 that expressed the need for newer linux headers.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
    • built dpdk with libbpf 0.6 on Linux: it failed before due to old headers
    • built pkgsCross.gnu64.linuxHeaders on macOS: the result looks ok
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
@orivej orivej requested review from LnL7, Ericson2314 and alyssais Feb 1, 2020
# For some reason, doing `make install_headers` twice, first without
# INSTALL_HDR_PATH=$out then with, is neccessary to get this to work
# for darwin cross. @Ericson2314 has no idea why.
+ ''

This comment has been minimized.

@orivej

orivej Feb 1, 2020
Author Contributor

Double make headers_install is not needed any more.

+ ''
find "$out" -name '..install.cmd' -print0 | xargs -0 rm

This comment has been minimized.

@orivej

orivej Feb 1, 2020
Author Contributor

The new make headers_install copies only .h files.

@@ -1,32 +1,12 @@
diff --git a/Makefile b/Makefile

This comment has been minimized.

@orivej

orivej Feb 1, 2020
Author Contributor

This patch may not be necessary any more, but I'm not sure.

This comment has been minimized.

@orivej

orivej Feb 1, 2020
Author Contributor

I have checked that deleting this patch has no effect on the result of pkgsCross.gnu64.linuxHeaders, so I've deleted it.

@@ -239,7 +239,7 @@ ifdef CONFIG_RETPOLINE
endif

@@ -231,3 +231,3 @@ endif
archscripts: scripts_basic
- $(Q)$(MAKE) $(build)=arch/x86/tools relocs
+ $(Q)$(MAKE) $(build)=arch/x86/tools

This comment has been minimized.

@orivej

orivej Feb 1, 2020
Author Contributor

This is still needed, otherwise compilation fails without elf.h.

@@ -16,7 +16,7 @@ let
# We do this so we have a build->build, not build->host, C compiler.
depsBuildBuild = [ buildPackages.stdenv.cc ];
# `elf-header` is null when libc provides `elf.h`.
nativeBuildInputs = [ perl elf-header ];
nativeBuildInputs = [ perl rsync elf-header ];

This comment has been minimized.

@orivej

orivej Feb 1, 2020
Author Contributor

The new make headers_install installs the headers using rsync (to copy only .h files).

Linux has reworked its header install process in
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=59b2bd05f5f4dc62979c2e82ddd384f07e8f10bc
It has introduced `make headers` that prepares the headers in ./usr/include, and
made `make headers_install` copy the headers from there using rsync.
@orivej orivej force-pushed the orivej:linux-headers branch from d45032c to d48e99d Feb 1, 2020
@FRidh FRidh added this to WIP in Staging via automation Feb 5, 2020
@FRidh FRidh added this to the 20.03 milestone Feb 5, 2020
@orivej-nixos orivej-nixos moved this from WIP to Needs review in Staging Feb 5, 2020
@orivej
Copy link
Contributor Author

@orivej orivej commented Feb 5, 2020

I have checked that there is no difference between 5.5 and 5.5.2 (except in the value of LINUX_VERSION_CODE in linux/version.h), and in general as I understand there should be no difference across all 5.5.* series, therefore we should not update the headers from 5.5 to 5.5.x.

@thefloweringash
Copy link
Member

@thefloweringash thefloweringash commented Feb 9, 2020

This seems to break the bootstrap for aarch64:

$ nix-build -A hello -j1
these derivations will be built:
 [...]
building '/nix/store/d5fhzi0dvfdqcjbawv21bhs8pnifxpfl-popt-1.16.drv'...
unpacking sources
unpacking source archive /nix/store/vd3nbw7w1d7yrb89c3vj5kwk3yb8sgf1-popt-1.16.tar.gz
source root is popt-1.16
setting SOURCE_DATE_EPOCH to timestamp 1273006578 of file popt-1.16/test-poptrc
patching sources
configuring
fixing libtool script ./ltmain.sh
configure flags: --disable-static --disable-dependency-tracking --prefix=/nix/store/88izzq2bqs47q7a9q9q7ahl9yi7l6p7c-popt-1.16
checking build system type... ./config.guess: unable to guess system type

This script, last modified 2009-11-20, has failed to recognize
the operating system you are using. It is advised that you
download the most up to date version of the config scripts from

  http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD
and
  http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD

If the version you run (./config.guess) is already up to date, please
send the following data and any information you think might be
pertinent to <config-patches@gnu.org> in order to provide the needed
information to handle your system.

config.guess timestamp = 2009-11-20

uname -m = aarch64
uname -r = 4.19.68
uname -s = Linux
uname -v = #1-NixOS SMP PREEMPT Thu Jan 1 00:00:01 UTC 1970

/usr/bin/uname -p =
/bin/uname -X     =

hostinfo               =
/bin/universe          =
/usr/bin/arch -k       =
/bin/arch              =
/usr/bin/oslevel       =
/usr/convex/getsysinfo =

UNAME_MACHINE = aarch64
UNAME_RELEASE = 4.19.68
UNAME_SYSTEM  = Linux
UNAME_VERSION = #1-NixOS SMP PREEMPT Thu Jan 1 00:00:01 UTC 1970
configure: error: cannot guess build type; you must specify one
builder for '/nix/store/d5fhzi0dvfdqcjbawv21bhs8pnifxpfl-popt-1.16.drv' failed with exit code 1
cannot build derivation '/nix/store/awz6s24220mxm03dqq3p61hn05x51mib-rsync-3.1.3.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/jjnrjx8qazviqvlg33i5b68jd0lagkdr-linux-headers-5.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/rlkn045ii7sxp91ly4whpbq3ks410kad-glibc-2.27.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/zrxffnf7iydi0ahqxl91jj653p9cc9x7-stdenv-linux.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/wi7kwn2hkghzf8p5ygcx008gz5h3bbj9-hello-2.10.drv': 1 dependencies couldn't be built
error: build of '/nix/store/wi7kwn2hkghzf8p5ygcx008gz5h3bbj9-hello-2.10.drv' failed
@orivej
Copy link
Contributor Author

@orivej orivej commented Feb 10, 2020

I can not reproduce this with nix-build . -A pkgsCross.aarch64-multiplatform.hello and I don't have access to an aarch64 machine to perform a native build. Could you save the build directory after nix-build -A hello --keep-failed and attach it as a tar.gz to this issue? It may also be worth to check if this update is the cause by testing it on top of master (git checkout master; git cherry-pick -n d48e99d9122119ce1741867b5a7ea682d134f713; nix-build . -A hello).

@worldofpeace worldofpeace removed this from the 20.03 milestone Feb 10, 2020
@thefloweringash
Copy link
Member

@thefloweringash thefloweringash commented Feb 11, 2020

I think the problem is that rsync is now being required by an earlier stage during stdenv bootstrap. linuxHeaders is produced at stage2, so stage2 must be able to build rsync and its popt dependency.

Starting from stage3 onwards there's a hook that updates outdated config.guess files, which matches the error message from popt's configuration failure. Taking this hook and trivially applying it to stage2 is sufficient for me to nix-build -A hello on aarch64-linux. I also tested armv7l-linux, it was able to produce a linux headers 5.5, but glibc is currently broken so I couldn't build a full stdenv.

diff --git a/pkgs/stdenv/linux/default.nix b/pkgs/stdenv/linux/default.nix
index 60c0730dce1..cf8705bd891 100644
--- a/pkgs/stdenv/linux/default.nix
+++ b/pkgs/stdenv/linux/default.nix
@@ -255,6 +255,10 @@ in
         libc = getLibc self;
       };
     };
+    extraNativeBuildInputs = [ prevStage.patchelf ] ++
+      # Many tarballs come with obsolete config.sub/config.guess that don't recognize aarch64.
+      lib.optional (!localSystem.isx86 || localSystem.libc == "musl")
+                   prevStage.updateAutotoolsGnuConfigScriptsHook;
   })

(diff generated after cherry-picking onto master @ 6932a75)

@orivej
Copy link
Contributor Author

@orivej orivej commented Feb 11, 2020

That looks like a useful change, but since it is trivial to avoid rsync and this reduces the amount of packages needed to reach stage3, I have done just that. (I have checked that the result does not change.)

@orivej-nixos orivej-nixos merged commit b39c52f into NixOS:staging Feb 11, 2020
16 checks passed
16 checks passed
linuxHeaders on x86_64-darwin No attempt
Details
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A darwin-tested
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release-combined.nix -A tested
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
linuxHeaders on aarch64-linux Success
Details
linuxHeaders on x86_64-linux Success
Details
Staging automation moved this from Needs review to Done Feb 11, 2020
@orivej
Copy link
Contributor Author

@orivej orivej commented Feb 20, 2020

This update caused a few failures (not in master yet; based on this eval diff, checkboxes for fixes):

  • lockdep — error: unknown type name '__aligned_u64' — maybe it defines __ASSEMBLY__?
  • dnsmasq — error: 'SIOCGSTAMP' undeclared — probably does not include linux/sockios.h
  • can-utils — same as dnsmasq
  • dpdk — error: 'struct xsk_umem_config' has no member named 'flags' — it assumes that if linux-headers are recent than libbpf is recent; we shall update libbpf
  • iptraf — error: linux/isdn.h: No such file or directory — isdn.h is gone since Linux 5.3
  • klibc — error: 'MAP_PRIVATE' undeclared — probably does not include linux/mman.h

(discovered later)

@vcunat
Copy link
Member

@vcunat vcunat commented Feb 20, 2020

dnsmasq was broken on master and I fixed it in 482642a (and 944775e).

@orivej
Copy link
Contributor Author

@orivej orivej commented Feb 20, 2020

Ok, this has reached master yesterday (2020-02-19 08:18 UTC).

@orivej
Copy link
Contributor Author

@orivej orivej commented Feb 20, 2020

I thought that this is a good candidate for 20.03 since it defaults to Linux 5.4 and the update will allow applications to use newer kernel features; but its backport depends on backporting fixes for the packages that are broken by the update.

@Ericson2314
Copy link
Member

@Ericson2314 Ericson2314 commented Feb 20, 2020

Thanks @orivej for checking all the thorny cross cases!

vcunat added a commit that referenced this pull request Feb 20, 2020
vcunat added a commit that referenced this pull request Feb 20, 2020
dtzWill added a commit to dtzWill/nixpkgs that referenced this pull request Feb 21, 2020
(cherry picked from commit f04825c)
dtzWill added a commit to dtzWill/nixpkgs that referenced this pull request Feb 21, 2020
(cherry picked from commit 627a9ff)
@alyssais
Copy link
Member

@alyssais alyssais commented Feb 22, 2020

Why 5.5, and not 5.4 (LTS)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Staging
  
Done
Linked issues

Successfully merging this pull request may close these issues.

None yet

8 participants
You can’t perform that action at this time.