Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to override glibc with a patch for a specific package? #50329

Open
nh2 opened this issue Nov 13, 2018 · 18 comments
Open

How to override glibc with a patch for a specific package? #50329

nh2 opened this issue Nov 13, 2018 · 18 comments
Labels
6.topic: stdenv Standard environment

Comments

@nh2
Copy link
Contributor

nh2 commented Nov 13, 2018

My Haskell app calls malloc_info() for online memory usage reporting.

I have patched a bug in glibc that makes some values show up as always 0.

I would like to build only my Haskell package so that it uses the patched glibc (so that I don't have to rebuild all the other packages, like Chromium, that my package merely execve()s).

I'm considering:

  unmodified_nixpkgs = import <nixpkgs> {};
  nixpkgs = import <nixpkgs> {
    config = {

      packageOverrides = pkgs: rec {

        glibc = pkgs.glibc.overrideAttrs (old: {
          patches = [
            ./malloc_info-Fix-missing-accounting-of-top-chunk.patch
          ];
        });

But my Haskell package doesn't seem to pick up this glibc override; it is not compiled at all.

@cleverca22 thinks it's due to

boot = import ../stdenv/booter.nix { inherit lib allPackages; };
stages = stdenvStages {
inherit lib localSystem crossSystem config overlays;
};
pkgs = boot stages;

i think the boot on line 91 is where it makes the main stdenv, and line 94 is where your overrides come into play, too late

How can I make an override, overlay, or otherwise, to have my package use my patched glibc?

@nh2
Copy link
Contributor Author

nh2 commented Nov 13, 2018

CC @Ericson2314 @dtzWill as you may know this from the cross compilation efforts.

@coretemp
Copy link
Contributor

According to the documentation of overlays you should be able to use overlays. Now, if that doesn't work, this shouldn't be a question, but it should be filed as a bug.

@matthewbauer
Copy link
Member

I think you will need to do it by overriding stdenv. You should be able to do this with stdenvOverrides:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/stage.nix#L99-L103

But it will be a mass rebuild causing because GCC needs glibc. With my PR here:

#48803

You should be able to do what you are asking without a mass rebuild using crossOverlays. 😄

@nh2
Copy link
Contributor Author

nh2 commented Nov 13, 2018

You should be able to do this with stdenvOverrides

@matthewbauer I'm not familiar with that, how do I do it? Do I stick stdenvOverrides somewhere like I do with config.packageOverrides?

(mass rebuild is OK to me, since it'll be confided to everything built on top of the so-modified nixpkgs import)

@matthewbauer
Copy link
Member

matthewbauer commented Nov 13, 2018

Maybe you don't need stdenvOverrides, actually. I think something like this should work:

self: super: {
  libc = super.glibc.override {
    inherit (super) stdenv;
  };
  linuxHeaders = (super.callPackage "${super.path}/pkgs/os-specific/linux/kernel-headers" {
    inherit (super) stdenv;
  }).linuxHeaders;

  stdenv = if super.stdenv.isLinux && super.stdenv.cc != null && super.stdenv.cc ? override then super.stdenv.override {
    cc = super.stdenv.cc.override {
      inherit (self) libc;
      bintools = if super.stdenv.cc.bintools ? override then
        (super.stdenv.cc.bintools.override {
          inherit (self) libc;
        }) else super.stdenv.cc.bintools;
    };
  } else super.stdenv;
}

@nh2
Copy link
Contributor Author

nh2 commented Nov 29, 2018

Hmm, that gets me error: infinite recursion encountered, at undefined position (note I'm based on 18.03, should it work also there?).

@nh2
Copy link
Contributor Author

nh2 commented Nov 29, 2018

Here's my error:

error: while evaluating anonymous function at nixpkgs/pkgs/top-level/impure.nix:15:1, called from undefined position:
while evaluating anonymous function at nixpkgs/pkgs/top-level/default.nix:20:1, called from nixpkgs/pkgs/top-level/impure.nix:82:1:
while evaluating anonymous function at nixpkgs/pkgs/stdenv/booter.nix:42:1, called from nixpkgs/pkgs/top-level/default.nix:97:10:
while evaluating 'dfold' at nixpkgs/pkgs/stdenv/booter.nix:60:27, called from nixpkgs/pkgs/stdenv/booter.nix:117:4:
while evaluating 'go' at nixpkgs/pkgs/stdenv/booter.nix:63:18, called from nixpkgs/pkgs/stdenv/booter.nix:72:13:
while evaluating 'folder' at nixpkgs/pkgs/stdenv/booter.nix:89:33, called from nixpkgs/pkgs/stdenv/booter.nix:68:18:
while evaluating 'allPackages' at nixpkgs/pkgs/top-level/default.nix:87:17, called from nixpkgs/pkgs/stdenv/booter.nix:101:10:
while evaluating anonymous function at nixpkgs/pkgs/top-level/stage.nix:12:1, called from nixpkgs/pkgs/top-level/default.nix:87:26:
while evaluating 'fix' at nixpkgs/lib/fixed-points.nix:19:9, called from nixpkgs/pkgs/top-level/stage.nix:136:3:
while evaluating 'extends' at nixpkgs/lib/fixed-points.nix:44:24, called from nixpkgs/lib/fixed-points.nix:19:20:
while evaluating 'stdenvOverrides' at nixpkgs/pkgs/top-level/stage.nix:105:27, called from nixpkgs/lib/fixed-points.nix:44:67:
while evaluating the attribute 'stdenv.overrides' at nixpkgs-overlays/default.nix:59:3:
while evaluating the attribute 'stdenv.isLinux' at nixpkgs/pkgs/top-level/stage.nix:85:5:
while evaluating the attribute 'stdenv' at nixpkgs/pkgs/top-level/stage.nix:77:12:
while evaluating the attribute 'stdenv' at nixpkgs/pkgs/stdenv/linux/default.nix:312:5:
while evaluating 'makeOverridable' at nixpkgs/lib/customisation.nix:72:24, called from nixpkgs/pkgs/stdenv/linux/default.nix:312:14:
while evaluating anonymous function at nixpkgs/pkgs/stdenv/generic/default.nix:3:1, called from nixpkgs/lib/customisation.nix:74:12:
while evaluating 'optionalAttrs' at nixpkgs/lib/attrsets.nix:321:25, called from nixpkgs/pkgs/stdenv/generic/default.nix:67:5:
while evaluating 'optional' at nixpkgs/lib/lists.nix:198:20, called from nixpkgs/pkgs/stdenv/linux/default.nix:356:16:
while evaluating 'go' at nixpkgs/pkgs/stdenv/booter.nix:63:18, called from nixpkgs/pkgs/stdenv/booter.nix:69:18:
while evaluating 'folder' at nixpkgs/pkgs/stdenv/booter.nix:89:33, called from nixpkgs/pkgs/stdenv/booter.nix:68:18:
while evaluating 'allPackages' at nixpkgs/pkgs/top-level/default.nix:87:17, called from nixpkgs/pkgs/stdenv/booter.nix:101:10:
while evaluating anonymous function at nixpkgs/pkgs/top-level/stage.nix:12:1, called from nixpkgs/pkgs/top-level/default.nix:87:26:
while evaluating 'fix' at nixpkgs/lib/fixed-points.nix:19:9, called from nixpkgs/pkgs/top-level/stage.nix:136:3:
while evaluating 'extends' at nixpkgs/lib/fixed-points.nix:44:24, called from nixpkgs/lib/fixed-points.nix:19:20:
while evaluating 'stdenvOverrides' at nixpkgs/pkgs/top-level/stage.nix:105:27, called from nixpkgs/lib/fixed-points.nix:44:67:
while evaluating the attribute 'stdenv.overrides' at nixpkgs-overlays/default.nix:59:3:
while evaluating the attribute 'stdenv.cc' at nixpkgs/pkgs/stdenv/generic/default.nix:138:14:
while evaluating 'go' at nixpkgs/pkgs/stdenv/booter.nix:63:18, called from nixpkgs/pkgs/stdenv/booter.nix:69:18:
while evaluating 'folder' at nixpkgs/pkgs/stdenv/booter.nix:89:33, called from nixpkgs/pkgs/stdenv/booter.nix:68:18:
while evaluating 'allPackages' at nixpkgs/pkgs/top-level/default.nix:87:17, called from nixpkgs/pkgs/stdenv/booter.nix:101:10:
while evaluating anonymous function at nixpkgs/pkgs/top-level/stage.nix:12:1, called from nixpkgs/pkgs/top-level/default.nix:87:26:
while evaluating 'fix' at nixpkgs/lib/fixed-points.nix:19:9, called from nixpkgs/pkgs/top-level/stage.nix:136:3:
while evaluating 'extends' at nixpkgs/lib/fixed-points.nix:44:24, called from nixpkgs/lib/fixed-points.nix:19:20:
while evaluating 'stdenvOverrides' at nixpkgs/pkgs/top-level/stage.nix:105:27, called from nixpkgs/lib/fixed-points.nix:44:67:
while evaluating the attribute 'stdenv.overrides' at nixpkgs-overlays/default.nix:59:3:
while evaluating the attribute 'stdenv.cc' at nixpkgs/pkgs/stdenv/generic/default.nix:138:14:
while evaluating 'go' at nixpkgs/pkgs/stdenv/booter.nix:63:18, called from nixpkgs/pkgs/stdenv/booter.nix:69:18:
while evaluating 'folder' at nixpkgs/pkgs/stdenv/booter.nix:89:33, called from nixpkgs/pkgs/stdenv/booter.nix:68:18:
while evaluating 'allPackages' at nixpkgs/pkgs/top-level/default.nix:87:17, called from nixpkgs/pkgs/stdenv/booter.nix:101:10:
while evaluating anonymous function at nixpkgs/pkgs/top-level/stage.nix:12:1, called from nixpkgs/pkgs/top-level/default.nix:87:26:
while evaluating 'fix' at nixpkgs/lib/fixed-points.nix:19:9, called from nixpkgs/pkgs/top-level/stage.nix:136:3:
while evaluating 'extends' at nixpkgs/lib/fixed-points.nix:44:24, called from nixpkgs/lib/fixed-points.nix:19:20:
while evaluating 'stdenvOverrides' at nixpkgs/pkgs/top-level/stage.nix:105:27, called from nixpkgs/lib/fixed-points.nix:44:67:
while evaluating the attribute 'stdenv.overrides' at nixpkgs-overlays/default.nix:59:3:
while evaluating the attribute 'stdenv.cc' at nixpkgs/pkgs/stdenv/generic/default.nix:138:14:
while evaluating 'makeOverridable' at nixpkgs/lib/customisation.nix:72:24, called from nixpkgs/pkgs/stdenv/linux/default.nix:90:19:
while evaluating anonymous function at nixpkgs/pkgs/build-support/cc-wrapper/default.nix:8:1, called from nixpkgs/lib/customisation.nix:74:12:
while evaluating the attribute 'binutils' at nixpkgs/pkgs/stdenv/linux/default.nix:196:7:
while evaluating the attribute 'binutils_nogold' at nixpkgs/pkgs/top-level/all-packages.nix:7526:3:
while evaluating 'lowPrio' at nixpkgs/lib/meta.nix:52:13, called from nixpkgs/pkgs/top-level/all-packages.nix:7526:21:
while evaluating 'addMetaAttrs' at nixpkgs/lib/meta.nix:15:28, called from nixpkgs/lib/meta.nix:52:18:
while evaluating 'wrapBintoolsWith' at nixpkgs/pkgs/top-level/all-packages.nix:6890:22, called from nixpkgs/pkgs/top-level/all-packages.nix:7512:18:
while evaluating 'callPackageWith' at nixpkgs/lib/customisation.nix:113:35, called from nixpkgs/pkgs/top-level/all-packages.nix:6890:42:
while evaluating 'makeOverridable' at nixpkgs/lib/customisation.nix:72:24, called from nixpkgs/lib/customisation.nix:117:8:
while evaluating anonymous function at nixpkgs/pkgs/build-support/bintools-wrapper/default.nix:8:1, called from nixpkgs/lib/customisation.nix:74:12:
while evaluating the attribute 'cc.nativeTools' at nixpkgs/pkgs/stdenv/generic/default.nix:138:14:
while evaluating 'override' at nixpkgs/lib/customisation.nix:78:20, called from nixpkgs-overlays/default.nix:60:10:
while evaluating 'makeOverridable' at nixpkgs/lib/customisation.nix:72:24, called from nixpkgs/lib/customisation.nix:78:29:
while evaluating anonymous function at nixpkgs/pkgs/build-support/cc-wrapper/default.nix:8:1, called from nixpkgs/lib/customisation.nix:74:12:
while evaluating the attribute 'libc' at nixpkgs-overlays/default.nix:56:3:
while evaluating the attribute 'glibc.override' at nixpkgs/pkgs/top-level/all-packages.nix:8995:3:
while evaluating 'callPackageWith' at nixpkgs/lib/customisation.nix:113:35, called from nixpkgs/pkgs/top-level/all-packages.nix:8989:16:
while evaluating 'makeOverridable' at nixpkgs/lib/customisation.nix:72:24, called from nixpkgs/lib/customisation.nix:117:8:
while evaluating anonymous function at nixpkgs/pkgs/development/libraries/glibc/default.nix:1:1, called from nixpkgs/lib/customisation.nix:74:12:
while evaluating the attribute 'cc.isGNU' at nixpkgs/pkgs/stdenv/generic/default.nix:138:14:
infinite recursion encountered, at undefined position

@Ericson2314
Copy link
Member

This is going to suck until libc is just a regular library. Until then, overriding stdenv is the right approach.

@dguibert
Copy link
Member

@nh2 Have you a working stdenv with your patched glibc?
I also need a custom glibc version for a package, tried the example from @matthewbauer as well but don't figure out how to get an overriden stdenv. Any of my attemps to get it via an overlay lead to the inifinite recursion...

@nh2
Copy link
Contributor Author

nh2 commented May 27, 2019

@dguibert No, I didn't manage to get that, and used patchelf instead to replace the libc linked into the executable in e.g. postBuild.

@matthewbauer matthewbauer modified the milestones: 19.03, 19.09 May 27, 2019
@dguibert
Copy link
Member

Got it working with:

let
unmodified_nixpkgs = import <nixpkgs> {};
nixpkgs = import <nixpkgs> {
   overlays = [ (self: super: {
      glibc = ....;
   })];
  };
in nixpkgs

nixpkgs contains a stdenv with the modified glibc.

ps: the functions appendOverlays and extend are called after the stages of stdenv have been initialized. So the overriden packages are not in the bootstrap process. Maybe this applies to config.packageOverrides...

@matthewbauer matthewbauer modified the milestones: 19.09, 20.03 Aug 15, 2019
@thongpv87
Copy link

@dguibert Can you post the full overlays statement.

@dguibert
Copy link
Member

@thongpv3, here is the content of the overriden glibc

let
nixpkgs = import <nixpkgs> {
   overlays = [ (final: prev: {
     glibc = prev.glibc.overrideAttrs (old: {
        name = "my-patched-glibc-${old.version}";
        patches = [
          #https://patches-gcc.linaro.org/patch/21643/raw/
          ./malloc-Fix-missing-accounting-of-top-chunk-in-malloc_info-BZ-24026.diff
        ];
    });
  })];
};
in nixpkgs

@disassembler disassembler modified the milestones: 20.03, 20.09 Feb 5, 2020
@stale
Copy link

stale bot commented Aug 4, 2020

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Aug 4, 2020
@nh2
Copy link
Contributor Author

nh2 commented Aug 4, 2020

Still important to me.

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Aug 4, 2020
@FRidh FRidh modified the milestones: 20.09, 21.03 Dec 20, 2020
@stale
Copy link

stale bot commented Jun 18, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 18, 2021
@nh2
Copy link
Contributor Author

nh2 commented Jul 9, 2021

A bit related: #129595

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 9, 2021
@stale
Copy link

stale bot commented Jan 9, 2022

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 9, 2022
@Artturin Artturin modified the milestones: 21.05, 23.05 Dec 31, 2022
@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Dec 31, 2022
@Artturin Artturin removed this from the 23.05 milestone Apr 15, 2023
@Artturin Artturin added the 6.topic: stdenv Standard environment label Apr 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: stdenv Standard environment
Projects
None yet
Development

No branches or pull requests

9 participants