New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pkgs.overrideWithScope #44196

Open
wants to merge 1 commit into
base: master
from

Conversation

@nbp
Member

nbp commented Jul 29, 2018

Add pkgs.overrideWithScope. This function is similar to pkgs.override, except that it filters the arguments like callPackage does.

While this might seems like a small addition, this is actually a really really big change for the future of Nixpkgs. This function can help us simplify a lot the manipulation of emacsPackages, pythonPackages, haskellPackages, linuxPackages, ... This can also help us rewrite the bootstrapping done before all-packages and move it within all-packages.nix. and on a simpler note, this will help solving the #43755 as explained in #43828.

The reason which makes this function is a revolution for the implementation of package sets is that it make it simple to ""recursively"" (dependencies of dependencies of dependencies) and conherently override the input of a set of packages with a simple function, which does not appear in future overlay.

To make it clearer, here is a set of overlay:

import <nixpkgs> { overlays = [
  # Add an interpreter and its package set.
  (self: super: 
  let interpreterPackages = withSet: {
      foo = super.lib.callPackageWith withSet ./pkgs/interpreter/foo { /* interpreter libxml */ };
      bar = super.lib.callPackageWith withSet ./pkgs/interpreter/bar { /* interpreter foo */ };
      baz = super.lib.callPackageWith withSet ./pkgs/interpreter/baz { /* interpreter foo bar qt */ };
      qux = super.lib.callPackageWith withSet ./pkgs/interpreter/qux { /* interpreter bar baz */ };
    };
  in
  rec {
    interpreter27 = super.callPackage ./pkgs/interpreter_27.nix {};
    interpreter35 = super.callPackage ./pkgs/interpreter_35.nix {};
    interpreter27Packages = let withSet = self // { interpreter = interpreter27; } // self.interpreter27Packages; in
      super.lib.mapAttrs (n: v: v.overrideWithScope withSet) (interpreterPackages withSet);
    interpreter35Packages = let withSet = self // { interpreter = interpreter35; } // self.interpreter35Packages; in
      super.lib.mapAttrs (n: v: v.overrideWithScope withSet) (interpreterPackages withSet);
  })
  # Change a package dependency. Like any ordinary package, as opposed as today.
  (self: super: {
    interpreter27Packages = super.interpreter27Packages // {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
    };
  })
  # Duplicate a package set with a custom interpreter.
  (self: super: {
    interpreter27ProfilePackages = let withSet = self // self.interpreter27ProfilePackages; in
      super.lib.mapAttrs (n: v: v.overrideWithScope withSet) (super.interpreter27Packages // {
      interpreter = super.interpreter27.override { withProfiler = true; };
    });
  })
]; }

What is important to note, is that:

  • Changing a package within an existing attribute set, except for the attribute set manipulation, is like any ordinary package.
  • Creating a forked configuration (with a profiler for example), is as simple as changing the interpreter in a new set which is isolated with overrideWithScope.

To make it simple, pkgs.overrideWithScope make it possible to leverage the Nixpkgs fix-point without the burden of writing too many pkgs.override by hand (which need to have no-more than the expected set of arguments).

As it leverage the Nixpkgs fix-point, there is no need to unwrap the function to override packages within a given set, like is done with package sets today (Haskell being infamous due to the stacking of multiple fix-points).

Ideally, we would want to have the pkgs.overrideWithScope evaluated before the resolution of the arguments of a given package, which would not be possible until we add some post-processing of all packages at the end of Nixpkgs fix-point. This enforces that we have to provide the withSet argument to the interpreter set, and implies that we cannot have a raw set of packages which lives independently of a given interpreter version. (Note, one could cheat by providing a fake version of missing packages interpreter = null; libxml = null; qt = null;)

Another issue, which already exists but is made explicit here, is that self // self.interpreterPackages is inefficient in terms of memory because attribute sets are not lazy. This does not worry me much as this is a fixable issue in Nixpkgs, as long as the expression is strictly used as argument of callPackagesWith and overrideWithScope

Other suggestions:

  • Renaming overrideWithScope to overrideWith.
  • Replacing override by overrideWithScope.
@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Jul 29, 2018

Member

I'm not quite sure about this idiom. Say we had this function:

let
  # injectOverride
  # :  (name: String)
  # -> ({ a } -> { a } -> { a })
  # -> ({ name: {a}; ..b } -> { name: {a}; ..b } -> { name: {a}; ..b })
  injectOverride = name: f: self: super: { 
    "${name}" = super.${name} // f self.${name} super.${self} // {
      __overlay__ = lib.composeExtensions (super.__inner__ or (_: _: {})) f;
    };
  }; 

Then we can just do:

import <nixpkgs> { overlays = [
  # Add an interpreter and its package set.
  (self: super: 
  let 
    pkgs = self;
    interpreterPackages = self: {
      callPackage = pkgs.newScope self;
      foo = self.callPackage ./pkgs/interpreter/foo { /* interpreter libxml */ };
      bar = self.callPackage ./pkgs/interpreter/bar { /* interpreter foo */ };
      baz = self.callPackage ./pkgs/interpreter/baz { /* interpreter foo bar qt */ };
      qux = self.callPackage ./pkgs/interpreter/qux { /* interpreter bar baz */ };
    };
  in
  rec {
    interpreter27 = self.callPackage ./pkgs/interpreter_27.nix {};
    interpreter35 = self.callPackage ./pkgs/interpreter_35.nix {};
  } 
  // injectOverride "interpreter27Packages"
       (lib.extend (_: _: { interpreter = interpreter27; }) interpreterPackages)
  // injectOverride "interpreter35Packages"
       (lib.extend (_: _: { interpreter = interpreter35; }) interpreterPackages))
  # Change a package dependency. Like any ordinary package, as opposed as today.
  (injectOverride "interpreter27Packages" {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
  })
  # Duplicate a package set with a custom interpreter.
  (self: super: injectOverride "interpreter27ProfilePackages"
    super.interpreter27Packages.__overlay__ self super)
]; }
Member

Ericson2314 commented Jul 29, 2018

I'm not quite sure about this idiom. Say we had this function:

let
  # injectOverride
  # :  (name: String)
  # -> ({ a } -> { a } -> { a })
  # -> ({ name: {a}; ..b } -> { name: {a}; ..b } -> { name: {a}; ..b })
  injectOverride = name: f: self: super: { 
    "${name}" = super.${name} // f self.${name} super.${self} // {
      __overlay__ = lib.composeExtensions (super.__inner__ or (_: _: {})) f;
    };
  }; 

Then we can just do:

import <nixpkgs> { overlays = [
  # Add an interpreter and its package set.
  (self: super: 
  let 
    pkgs = self;
    interpreterPackages = self: {
      callPackage = pkgs.newScope self;
      foo = self.callPackage ./pkgs/interpreter/foo { /* interpreter libxml */ };
      bar = self.callPackage ./pkgs/interpreter/bar { /* interpreter foo */ };
      baz = self.callPackage ./pkgs/interpreter/baz { /* interpreter foo bar qt */ };
      qux = self.callPackage ./pkgs/interpreter/qux { /* interpreter bar baz */ };
    };
  in
  rec {
    interpreter27 = self.callPackage ./pkgs/interpreter_27.nix {};
    interpreter35 = self.callPackage ./pkgs/interpreter_35.nix {};
  } 
  // injectOverride "interpreter27Packages"
       (lib.extend (_: _: { interpreter = interpreter27; }) interpreterPackages)
  // injectOverride "interpreter35Packages"
       (lib.extend (_: _: { interpreter = interpreter35; }) interpreterPackages))
  # Change a package dependency. Like any ordinary package, as opposed as today.
  (injectOverride "interpreter27Packages" {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
  })
  # Duplicate a package set with a custom interpreter.
  (self: super: injectOverride "interpreter27ProfilePackages"
    super.interpreter27Packages.__overlay__ self super)
]; }
@nbp

This comment has been minimized.

Show comment
Hide comment
@nbp

nbp Jul 30, 2018

Member

@Ericson2314
The problem with the idiom you are suggesting is that users of a given sub-set of packages will have to know about the injectOverride function, and know when to use it. As opposed to overrideWithScope which would only be required at the creation/cloning of the sub-set.

(Also this should always be super.callPackage)

Honestly, if post-processing of Nixpkgs was an option, and every package would use callPackage (which is far from being the case), then we could make the callPackage function return a functor like object which is derived into a derivation only after going through the Nixpkgs fix-point, and thus we could go with just the following:

import <nixpkgs> { overlays = [
  # Add an interpreter and its package set.
  (self: super: 
  let interpreterPackages = {
      foo = super.callPackage ./pkgs/interpreter/foo { /* interpreter libxml */ };
      bar = super.callPackage ./pkgs/interpreter/bar { /* interpreter foo */ };
      baz = super.callPackage ./pkgs/interpreter/baz { /* interpreter foo bar qt */ };
      qux = super.callPackage ./pkgs/interpreter/qux { /* interpreter bar baz */ };
    };
  in
  rec {
    interpreter27 = super.callPackage ./pkgs/interpreter_27.nix {};
    interpreter35 = super.callPackage ./pkgs/interpreter_35.nix {};
    interpreter27Packages = { interpreter = interpreter27; } // interpreterPackages;
    interpreter35Packages = { interpreter = interpreter35; } // interpreterPackages;
  })
  # Change a package dependency. Like any ordinary package, as opposed as today.
  (self: super: {
    interpreter27Packages = super.interpreter27Packages // {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
    };
  })
  # Duplicate a package set with a custom interpreter.
  (self: super: {
    interpreter27ProfilePackages = super.interpreter27Packages // {
      interpreter = super.interpreter27.override { withProfiler = true; };
    });
  })
]; }

As the scope would be determine by the location of the package within Nixpkgs. But this future is not yet there, because not every package uses callPackage, and not every dependency comes from self.

Member

nbp commented Jul 30, 2018

@Ericson2314
The problem with the idiom you are suggesting is that users of a given sub-set of packages will have to know about the injectOverride function, and know when to use it. As opposed to overrideWithScope which would only be required at the creation/cloning of the sub-set.

(Also this should always be super.callPackage)

Honestly, if post-processing of Nixpkgs was an option, and every package would use callPackage (which is far from being the case), then we could make the callPackage function return a functor like object which is derived into a derivation only after going through the Nixpkgs fix-point, and thus we could go with just the following:

import <nixpkgs> { overlays = [
  # Add an interpreter and its package set.
  (self: super: 
  let interpreterPackages = {
      foo = super.callPackage ./pkgs/interpreter/foo { /* interpreter libxml */ };
      bar = super.callPackage ./pkgs/interpreter/bar { /* interpreter foo */ };
      baz = super.callPackage ./pkgs/interpreter/baz { /* interpreter foo bar qt */ };
      qux = super.callPackage ./pkgs/interpreter/qux { /* interpreter bar baz */ };
    };
  in
  rec {
    interpreter27 = super.callPackage ./pkgs/interpreter_27.nix {};
    interpreter35 = super.callPackage ./pkgs/interpreter_35.nix {};
    interpreter27Packages = { interpreter = interpreter27; } // interpreterPackages;
    interpreter35Packages = { interpreter = interpreter35; } // interpreterPackages;
  })
  # Change a package dependency. Like any ordinary package, as opposed as today.
  (self: super: {
    interpreter27Packages = super.interpreter27Packages // {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
    };
  })
  # Duplicate a package set with a custom interpreter.
  (self: super: {
    interpreter27ProfilePackages = super.interpreter27Packages // {
      interpreter = super.interpreter27.override { withProfiler = true; };
    });
  })
]; }

As the scope would be determine by the location of the package within Nixpkgs. But this future is not yet there, because not every package uses callPackage, and not every dependency comes from self.

@aszlig

aszlig approved these changes Aug 1, 2018

let
ff = f origArgs;
overrideWith = newArgs: origArgs // (if lib.isFunction newArgs then newArgs origArgs else newArgs);
intersectArgs = if fnArgs != {} then builtins.intersectAttrs fnArgs else (a: a);
overrideWithScope = newScope: overrideWith (intersectArgs newScope);

This comment has been minimized.

@aszlig

aszlig Aug 1, 2018

Member

Very cool, now I can finally get rid of a bit of churn in some of my Nix expressions, thanks :-)

One minor nitpick however:

If my assumption is correct that newScope should always be an attribute set (intersectAttrs would bail out if not), there is no need to go through isFunction in overrideWith above again, so removing intersectArgs and changing overrideWithScope to the following would avoid that:

newScope: origArgs // (if fnArgs != {} then builtins.intersectAttrs fnArgs newScope else newScope)
@aszlig

aszlig Aug 1, 2018

Member

Very cool, now I can finally get rid of a bit of churn in some of my Nix expressions, thanks :-)

One minor nitpick however:

If my assumption is correct that newScope should always be an attribute set (intersectAttrs would bail out if not), there is no need to go through isFunction in overrideWith above again, so removing intersectArgs and changing overrideWithScope to the following would avoid that:

newScope: origArgs // (if fnArgs != {} then builtins.intersectAttrs fnArgs newScope else newScope)
/* `makeOverridable` takes a function from attribute set to attribute set, a
list of expected arguments and injects `override`, `overrideWithScope` and
`overrideDerivation` attibutes in the result of the function which can be
used to re-call the function with an overrided set of arguments.

This comment has been minimized.

@aszlig

aszlig Aug 1, 2018

Member

Also, just for the sake of completeness (this nitpick is so minor that it isn't even one), maybe add overrideWithScope in the output for the nix repl examples below.

@aszlig

aszlig Aug 1, 2018

Member

Also, just for the sake of completeness (this nitpick is so minor that it isn't even one), maybe add overrideWithScope in the output for the nix repl examples below.

@oxij

This comment has been minimized.

Show comment
Hide comment
@oxij

oxij Aug 1, 2018

Contributor
Contributor

oxij commented Aug 1, 2018

@twhitehead

This comment has been minimized.

Show comment
Hide comment
@twhitehead

twhitehead Aug 9, 2018

Contributor

For a bit now, I've been thinking it would be nice if the callPackage mechanism could add an override function that would do the equivalent of adding another overlay and then re-evaluate the package.

The idea would be then be that you could also effect changes in all the dependencies of a package as well. Typical examples would be where the dependent packages must also be changed or else you windup with the final executable sucking in two different versions of the same shared library (you see this in some existing expressions with the version of boost or cuda libraries used in its dependencies).

I originally thought this might be what this pull request was, but, upon closer looking, I don't believe so.
I thought I would mention it here though as it does seem there is some significant overlap.

Thanks! -Tyson

PS: Technically, I expect, it could make sense to only apply the overlay to runtime dependencies (i.e., not to the build time ones).

Contributor

twhitehead commented Aug 9, 2018

For a bit now, I've been thinking it would be nice if the callPackage mechanism could add an override function that would do the equivalent of adding another overlay and then re-evaluate the package.

The idea would be then be that you could also effect changes in all the dependencies of a package as well. Typical examples would be where the dependent packages must also be changed or else you windup with the final executable sucking in two different versions of the same shared library (you see this in some existing expressions with the version of boost or cuda libraries used in its dependencies).

I originally thought this might be what this pull request was, but, upon closer looking, I don't believe so.
I thought I would mention it here though as it does seem there is some significant overlap.

Thanks! -Tyson

PS: Technically, I expect, it could make sense to only apply the overlay to runtime dependencies (i.e., not to the build time ones).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment