max-jobs option is ignored

[B4dM4n]

Describe the bug

Running nix build on master ignores any max-jobs value set, either by nix.conf, --option max-jobs n or --max-jobs n.

Steps To Reproduce

1. Run

nix build -j2 -f =(echo 'with import <nixpkgs> { }; with lib; map (n: runCommand "test-${n}" { } "sleep 10") (genList toString 50)')

2. number of nixbld users builds run at the same time

or

nix build --store /tmp/nix-store -j2 -f =(echo 'with import <nixpkgs> { }; with lib; map (n: runCommand "test-${n}" { } "sleep 10") (genList toString 50)')

2. All builds run at the same time

Expected behavior

Only two parallel builds should run.

nix-env --version output

nix-env (Nix) 2.4pre20200626_b481d8a ( B4dM4n/nix@b481d8a current flakes branch + minor change)

Additional context

Building via nix-daemon will start one build for every nixbld user, otherwise there is no limit on the number of jobs started.

https://hydra.nixos.org/eval/1588015 is the first build I could find that show this behavior, which would point to #3564 introducing this.

[SFrijters]

Seems I'm running into this problem now as well.

[mstone]

I am also experiencing this bug, I think, on

$ nix --version
nix (Nix) 3.0pre20200917_649d3aa

on Darwin.

Further, building texlive now takes far too long (i.e., does not complete in many minutes, on a single-user install), I think due contention resulting from hundreds or thousands of builder processes being spawned.

/cc @balsoft

[balsoft]

Oh, sorry for breaking it :P Thanks for the ping, @mstone!

I have taken a quick look at it and I think I know how to fix it; I will try to find some time, but can't promise anything.

[mstone]

@balsoft -- not to worry, and thanks for taking a look at it!

Two followup notes:

1. while I believe I'm seeing the same issue that @B4dM4n described, I haven't done any significant work yet to confirm their analysis / bisection of which commit introduced the problem. As a result, if you find that you've hit a dead end while researching, please let me know and maybe we can explore the issue together -- I just pinged you because I wanted to make sure that you had seen @B4dM4n's issue report.

2. also, independent of all of the discussion above, congratulations on getting your first nix PR merged!

[balsoft]

Hopefully fixed in #4064