stdenv's linuxHeaders (or allowedRequisites) incorrect in the presence of overlays overriding glibc

[TravisWhitaker]

Consider the following expression, where we apply an overlay that uses an older NixOS release's glibc with newer nixpkgs (the motivation for such an overlay might be building DSOs that may be linked and loaded on some other Linux distribution).

let recentNixpkgsSrc = builtins.fetchGit
    {
        url = "https://github.com/nixos/nixpkgs";
        ref = "master";
        rev = "7c7d71d90e7d715847809a65fa2463c668e58ab3";
    };
    olderNixpkgsSrc = builtins.fetchGit
    {
        url = "https://github.com/nixos/nixpkgs";
        ref = "nixos-19.09";
        rev = "75f4ba05c63be3f147bcc2f7bd4ba1f029cedcb1";
    };
    oldPkgs = import olderNixpkgsSrc {};
    overlay = self: super: rec
    {
        glibc = oldPkgs.glibc // {pname = "glibc";};
        glibcLocales = oldPkgs.glibcLocales;
        glibcIconv = oldPkgs.glibcIconv;
        stdenv = super.stdenv //
        {
            overrides = self2: super2: super.stdenv.overrides self2 super2 //
            {
                glibc = glibc;
            };
        };
    };
in import recentNixpkgsSrc {overlays = [overlay];}

Based on https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/linux/default.nix, I'd expect to get a working stdenv with the older glibc (assuming that glibc 2.27 works with current master's toolchain, and it happens to for the revs I've chosen) and I do, until I get to the allowedRequisites check

$ nix-build ./repro.nix -A hello
...
building '/nix/store/n28xd4xq6f11lcbc1ir5lszc2bfj4n23-stdenv-linux.drv'...
output '/nix/store/cmk1rwz6raj480fqvv7rvxcabdmqh9k6-stdenv-linux' is not allowed to refer to the following paths:
  /nix/store/b3vajivl1gvq1c2p1x62672sqg8f983l-linux-headers-4.19.16

Initially one might try to fix this by getting linuxHeaders from the older nixpkgs too:

    overlay = self: super: rec
    {
        glibc = oldPkgs.glibc // {pname = "glibc";};
        glibcLocales = oldPkgs.glibcLocales;
        glibcIconv = oldPkgs.glibcIconv;
        linuxHeaders = oldPkgs.linuxHeaders;
        stdenv = super.stdenv //
        {
            overrides = self2: super2: super.stdenv.overrides self2 super2 //
            {
                glibc = glibc;
                linuxHeaders = linuxHeaders;
            };
        };

However, this doesn't work, because pkgs.linuxHeaders is somehow not the same linuxHeaders that's passed as input to glibc:

Welcome to Nix version 2.3.16. Type :? for help.

nix-repl> pkgs = import ./repro.nix

nix-repl> pkgs.linuxHeaders
«derivation /nix/store/n3vbnnrggh7prji14g1g2iizdjh06afq-linux-headers-4.19.16.drv»

nix-repl> pkgs.glibc.buildInputs
[ «derivation /nix/store/z0bf9m7i2228smqc97akym9ps6a0jrd7-linux-headers-4.19.16.drv» ]

The way allowedRequisites is computed in stdenv-linux suggests that stdenv needs to be given the same linuxHeaders that was used to build glibc, so this (totally cursed) overlay actually works:

    overlay = self: super: rec
    {
        glibc = oldPkgs.glibc // {pname = "glibc";};
        glibcLocales = oldPkgs.glibcLocales;
        glibcIconv = oldPkgs.glibcIconv;
        stdenv = super.stdenv //
        {
            overrides = self2: super2: super.stdenv.overrides self2 super2 //
            {
                glibc = glibc;
                linuxHeaders = builtins.head glibc.buildInputs;
            };
        };

$ ldd $(nix-build ./repro.nix -A hello)/bin/hello
        linux-vdso.so.1 (0x00007ffd48462000)
        libc.so.6 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libc.so.6 (0x00007fcefe42e000)
        /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fcefe3bd000)

So it seems to me, one of these must be true:

• This bit here https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/linux/default.nix#L421 is grabbing the wrong linuxHeaders if stdenvOverrides are used (although I can't work out how the stdenv bootstrap staging machinery works, so I'm not sure what's wrong).
• If stdenv must be passed the same linuxHeaders that's passed to glibc (and I assume that linuxHeaders that's different from top-level pkgs.linuxHeaders comes from inside the bootstrap machinery somewhere), shouldn't it grab it from glibc somehow (hopefully through a passthru and not the abomination builtins.head glibc.buildInputs?
• The most likely answer: I've misunderstood something about how stdenvOverrides/the stdenv bootstrapping business works.

[TravisWhitaker]

@Ericson2314 do you think there's a bug here? Or is the above the desired behavior of stdenvOverrides and/or stdenv/linux?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/derivation-that-builds-standard-linux-binaries/21557/7

[cameron-martin]

You can use linuxHeaders = glibc.linuxHeaders; instead of linuxHeaders = builtins.head glibc.buildInputs;. However, when I try this I get the following error:

error: assertion '(libc_bin == (bintools).libc_bin)' failed

       at /nix/store/wdf2jhfwsn294yk4n91fhz660bi9l0np-source/pkgs/build-support/cc-wrapper/default.nix:119:1:

          118| # Ensure bintools matches
          119| assert libc_bin == bintools.libc_bin;
             | ^
          120| assert libc_dev == bintools.libc_dev;
(use '--show-trace' to show detailed location information)

Do you know how to make these versions match?