nix-shell into default.nix: 'with import <nixpkgs> {};' ignored, but the same given on cmdline worked

[rene-tobner]

Describe the bug

nix-shell into default.nix: 'with import {};' ignored, but the same given on cmdline worked

Steps To Reproduce

default.nix:

with import <nixpkgs> {};
# same with:
#let
#  pkgs = import <nixpkgs> {};
#  inherit (pkgs) lib stdenv fetchFromGithub pkg-config xorg;
#in

{ lib, stdenv, fetchFromGitHub, pkg-config, xorg }:

[...]

1. Just using nix-shell on the above: lib's value non-existant

Expected behavior

Expected bahavior was seen with:

nix-shell -E 'with import <nixpkgs> {}; callPackage ./default.nix {}'

-> lib and the other references have their value and nix-shell is available!

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 4.19.0-17-amd64, Debian GNU/Linux, 10 (buster), nobuild`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.8.0`
 - channels(rtb): `"nixpkgs"`
 - nixpkgs: `/home/rtb/.nix-defexpr/channels/nixpkgs`

[jtojnar]

You are not doing the same. Notice the callPackage in the command-line version (omitting the nix-shell -E for simplicity):

with import <nixpkgs> {}; callPackage ./default.nix {}

What callPackage does is that it passes packages from the scope it belongs to as arguments to the function given to it as the first argument (or, if the argument is file path, to the function obtained by importing the path).

Avoiding the with statement to make it clear where do things come from, you have the following:

let
  pkgs = import <nixpkgs> {};
in
pkgs.callPackage ./default.nix {}

which is the same as:

let
  pkgs = import <nixpkgs> {};
in
pkgs.callPackage (import ./default.nix) {}

which further expands to:

let
  pkgs = import <nixpkgs> {};
in
pkgs.callPackage
  (
    with import <nixpkgs> {};
    { lib, stdenv, fetchFromGitHub, pkg-config, xorg }:

    /* ... */
  )
  {}

Assuming every used variable is available as one of the arguments, the with statement is useless here, since anything brought into the scope by it will be shadowed – remember Nix is a programming language and you generally cannot use “variables” that are not in scope.

let
  pkgs = import <nixpkgs> {};
in
pkgs.callPackage
  (
    { lib, stdenv, fetchFromGitHub, pkg-config, xorg }:

    /* ... */
  )
  {}

Now we can see substitute out callPackage:

let
  pkgs = import <nixpkgs> {};
in
(
  { lib, stdenv, fetchFromGitHub, pkg-config, xorg }:

  /* ... */
)
  (
    {
      lib = pkgs.lib;
      stdenv = pkgs.stdenv;
      fetchFromGitHub = pkgs.fetchFromGitHub;
      pkg-config = pkgs.pkg-config;
      xorg = pkgs.xorg;
    }
    // {} # This is the second argument of `callPackage`
  )

---

Whereas, if you just call nix-shell without arguments, it will be essentially the same as nix-shell -E 'import ./shell.nix' or nix-shell -E 'import ./default.nix' (if shell.nix does not exist).

Resolving

import ./default.nix

you get

with import <nixpkgs> {};

{ lib, stdenv, fetchFromGitHub, pkg-config, xorg }:

/* ... */

The with statement will make the packages from Nixpkgs available in the scope and return the inner expression. And since the inner expression is a function, nix-shell will try to call it. But you did not pass it any value, nor do the function arguments have default values so the call will fail:

error: cannot evaluate a function that has an argument without a value ('lib')

       Nix attempted to evaluate a function as a top level expression; in
       this case it must have its arguments supplied either by default
       values, or passed explicitly with '--arg' or '--argstr'. See
       https://nixos.org/manual/nix/stable/expressions/language-constructs.html#functions.

       at /home/jtojnar/Projects/nixpkgs/shell.nix:3:3:

            2|
            3| { lib, stdenv, fetchFromGitHub, pkg-config, xorg }:
             |   ^
            4|

[rene-tobner]

The thing is: I'm pretty sure, the other day my default.nix was working without trouble VS. you basically saying it's impossible to run my default.nix without callPackage ?!

Need some more time to digest your post, though.

By the way, my default.nix is the one in nixpkgs from spectrwm, with-statement added on top of it. And, I know I build it from source nix-shelling into the project, asaid default.nix. I can show the executable with nix-store-paths in it ...

[rene-tobner]

So, how to call the function in my default.nix and not from command line?

When I try to use callPackage inside the file, I get:

error: syntax error, unexpected ',', expecting '.' or '='

       at /home/rtb/code/spectrwm/default.nix:8:6:

            7| callpackage
            8| { lib, stdenv, fetchFromGitHub, pkg-config, xorg }:

and just passing pkgs directly after the function definition, like below, doesn't help either:

let
  pkgs = import <nixpkgs> {};
in

{ lib, stdenv, fetchFromGitHub, pkg-config, xorg }:

stdenv.mkDerivation {
  pname = "spectrwm";
  version = "3.4.1";
[...]
  meta = with lib; {
    description = "A tiling window manager";
    homepage    = "https://github.com/conformal/spectrwm";
    maintainers = with maintainers; [ christianharke ];
    license     = licenses.isc;
    platforms   = platforms.all;

    longDescription = ''
      spectrwm is a small dynamic tiling window manager for X11. It
      tries to stay out of the way so that valuable screen real estate
      can be used for much more important stuff. It has sane defaults
      and does not require one to learn a language to do any
      configuration. It was written by hackers for hackers and it
      strives to be small, compact and fast.
    '';
  };

} pkgs

[jtojnar]

it's impossible to run my default.nix without callPackage ?!

If you have a function, you must pass arguments it needs, just like in any other programming language. callPackage-based dependency injection is the best way for this but you can also pass the arguments manually as demonstrated in the “Now we can see substitute out callPackage” example above.

When I try to use callPackage inside the file, I get:

error: syntax error, unexpected ',', expecting '.' or '='

That is because you are missing parentheses around the function you want to pass as the first argument of callPackage (should also be camelCase BTW). Nix parser cannot really know where the function body should end¹, so it requires parentheses around functions used in certain places².

---

By the way, my default.nix is the one in nixpkgs from spectrwm, with-statement added on top of it. And, I know I build it from source nix-shelling into the project, asaid default.nix. I can show the executable with nix-store-paths in it ...

The store paths will be identical when the produced derivation is the same. It does not really matter how you pass around variables, as long as the same arguments are passed to mkDerivation.

The thing is: I'm pretty sure, the other day my default.nix was working without trouble VS. you basically saying it's impossible to run my default.nix without callPackage ?!

Well, if you have the function, you must pass it the arguments. But function arguments are not the only way to set “variables”. You can remove/comment out the function head and define the “variables” explicitly:

let
  pkgs = import <nixpkgs> {};
  lib = pkgs.lib;
  stdenv = pkgs.stdenv;
  fetchFromGitHub = pkgs.fetchFromGitHub;
  pkg-config = pkgs.pkg-config;
  xorg = pkgs.xorg;
in
# { lib, stdenv, fetchFromGitHub, pkg-config, xorg }:

/* ... */

Or you can introduce them implicitly using the with statement.

---

1. Consider simpler example f a: g a b, should b be an argument of f (parsed as f (a: g a) b) or g (parsed as f (a: g a b))?
2. Basically, any place where it would be ambiguous (e.g. function arguments). And for most expression types – really, the only expressions that do not require parentheses when passed as arguments are the single simple ones (various literals like numbers or strings, “variable” names, attribute reference foo.bar) and attribute sets… So when it sees { after callPackage the only allowed AST node corresponding to a code starting with { will be an attribute set.

[rene-tobner]

Are you already a computer science professor or a you just training? :)
Ok, I am on it soon...

[abhillman]

Related issue: #209635

[abhillman]

You are not doing the same. Notice the callPackage in the command-line version (omitting the nix-shell -E for simplicity):

with import <nixpkgs> {}; callPackage ./default.nix {}

What callPackage does is that it passes packages from the scope it belongs to as arguments to the function given to it as the first argument (or, if the argument is file path, to the function obtained by importing the path).

Avoiding the with statement to make it clear where do things come from, you have the following:

let
  pkgs = import <nixpkgs> {};
in
pkgs.callPackage ./default.nix {}

which is the same as:

let
  pkgs = import <nixpkgs> {};
in
pkgs.callPackage (import ./default.nix) {}

which further expands to:

let
  pkgs = import <nixpkgs> {};
in
pkgs.callPackage
  (
    with import <nixpkgs> {};
    { lib, stdenv, fetchFromGitHub, pkg-config, xorg }:

    /* ... */
  )
  {}

Assuming every used variable is available as one of the arguments, the with statement is useless here, since anything brought into the scope by it will be shadowed – remember Nix is a programming language and you generally cannot use “variables” that are not in scope.

let
  pkgs = import <nixpkgs> {};
in
pkgs.callPackage
  (
    { lib, stdenv, fetchFromGitHub, pkg-config, xorg }:

    /* ... */
  )
  {}

Now we can see substitute out callPackage:

let
  pkgs = import <nixpkgs> {};
in
(
  { lib, stdenv, fetchFromGitHub, pkg-config, xorg }:

  /* ... */
)
  (
    {
      lib = pkgs.lib;
      stdenv = pkgs.stdenv;
      fetchFromGitHub = pkgs.fetchFromGitHub;
      pkg-config = pkgs.pkg-config;
      xorg = pkgs.xorg;
    }
    // {} # This is the second argument of `callPackage`
  )

Whereas, if you just call nix-shell without arguments, it will be essentially the same as nix-shell -E 'import ./shell.nix' or nix-shell -E 'import ./default.nix' (if shell.nix does not exist).

Resolving

import ./default.nix

you get

with import <nixpkgs> {};

{ lib, stdenv, fetchFromGitHub, pkg-config, xorg }:

/* ... */

The with statement will make the packages from Nixpkgs available in the scope and return the inner expression. And since the inner expression is a function, nix-shell will try to call it. But you did not pass it any value, nor do the function arguments have default values so the call will fail:

error: cannot evaluate a function that has an argument without a value ('lib')

       Nix attempted to evaluate a function as a top level expression; in
       this case it must have its arguments supplied either by default
       values, or passed explicitly with '--arg' or '--argstr'. See
       https://nixos.org/manual/nix/stable/expressions/language-constructs.html#functions.

       at /home/jtojnar/Projects/nixpkgs/shell.nix:3:3:

            2|
            3| { lib, stdenv, fetchFromGitHub, pkg-config, xorg }:
             |   ^
            4|

@jtojnar -- if you're back to coming back to this -- I would expect the following to work in light of your comment, but I get a different error:

 % nix-shell -p '
let
  pkgs = import <nixpkgs> {};
in
pkgs.callPackage ./default.nix {}
'
error: Function called without required argument "findlib" at /Users/ahillman/Development/nixpkgs/pkgs/development/ocaml-modules/expat/default.nix:1, did you mean "indilib"?
(use '--show-trace' to show detailed location information)

[jtojnar]

I would expect the following to work in light of your comment, but I get a different error:

 % nix-shell -p '
let
  pkgs = import <nixpkgs> {};
in
pkgs.callPackage ./default.nix {}
'
error: Function called without required argument "findlib" at /Users/ahillman/Development/nixpkgs/pkgs/development/ocaml-modules/expat/default.nix:1, did you mean "indilib"?

pkgs.callPackage will only be able to fill in arguments that have corresponding attribute in pkgs. There is no pkgs.findlib – OCaml libraries are nested under ocamlPackages prefix.

If you are packaging an OCaml library (that will end up in pkgs/development/ocaml-modules), you should use pkgs.ocamlPackages.callPackage instead of pkgs.callPackage.

On the other hand, if you are packaging an OCaml program, you should take ocamlPackages as an argument and use ocamlPackages.findlib in the package expression (see e.g. frama-c).

Similar advice applies for other languages with libraries not in the top-level package set. For example, Python 3 packages are inside python3.pkgs attribute set.

[abhillman]

If you are packaging an OCaml library (that will end up in pkgs/development/ocaml-modules), you should use pkgs.ocamlPackages.callPackage instead of pkgs.callPackage.

Amazing -- thank you so much. Works perfectly:

nix-shell -p 'with (import <nixpkgs> {}); pkgs.ocamlPackages.callPackage ./default.nix {}'

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-get-latest-version-of-packages/33520/11