Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

haskell: tune generic-builder for more parallel build performance #86948

Merged
merged 1 commit into from May 7, 2020

Conversation

@peti
Copy link
Member

peti commented May 5, 2020

Raise the maximum number of cores used for parallel building from 4 to 16. Increase the size of the allocation area for GHC's garbage collector from 1 MiB to 64 MiB.

See https://www.twitch.tv/videos/611899011 for the motivation for this change.

@peti peti requested review from cdepillabout and nh2 May 5, 2020
Copy link
Contributor

nh2 left a comment

This looks sensible to me, but one question:

If it is done at the ghcWithPackages level, do/can/will we also have the ability to override it per Haskell package? I am suspecting that there might be certain large packages for which we may need to tune down -j or -A to not consume too much memory on builders that have less than the casual 3 TB RAM.

@peti peti mentioned this pull request May 5, 2020
2 of 3 tasks complete
@peti
Copy link
Member Author

peti commented May 7, 2020

It just occurred to me that the change to the ghcWithPackages wrapper is unnecessary, because you can do the same thing in ~/.cabal/config. Adding

program-default-options
  ghc-options: -j16 +RTS -A64M -RTS

there has the same effect -- at least if you're building with cabal-install. I'm pretty sure an identical mechanism exists for stack, too. So, on second thought I'd rather not introduce yet another location to configure default flags that's specific to Nixpkgs.

@peti peti force-pushed the peti:t/parallel-ghc-builds branch from 9c4ac9c to 7255d30 May 7, 2020
@peti peti changed the title ghcWithPackages: add ghcOptions argument to pass extra options to the compiler haskell: tune generic-builder for more parallel build performance May 7, 2020
@peti peti requested review from cdepillabout and nh2 May 7, 2020
Raise the maximum number of cores used for parallel building from 4 to 16.
Increase the size of the allocation area for GHC's garbage collector from 1 MiB
to 64 MiB.

See https://www.twitch.tv/videos/611899011 for the motivation for this change.
@peti peti force-pushed the peti:t/parallel-ghc-builds branch from 7255d30 to f6df169 May 7, 2020
@peti
Copy link
Member Author

peti commented May 7, 2020

I'll merge this change to haskell-updates so that we can get some test results from Hydra before the live stream tomorrow.

@peti peti merged commit 6a815f9 into NixOS:haskell-updates May 7, 2020
14 checks passed
14 checks passed
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="f6df169"; rev="f6df1697b750a614503aabae189c6f18d2eedb9f"; } ./pkgs/t
Details
grahamcofborg-eval-lib-tests nix-build --arg pkgs import ./. {} ./lib/tests/release.nix
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="f6df169"; rev="f6df1697b750a614503aabae189c6f18d2eedb9f"; } ./nixos/
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="f6df169"; rev="f6df1697b750a614503aabae189c6f18d2eedb9f"; } ./nixos/
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="f6df169"; rev="f6df1697b750a614503aabae189c6f18d2eedb9f"; } ./nixos/
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="f6df169"; rev="f6df1697b750a614503aabae189c6f18d2eedb9f"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="f6df169"; rev="f6df1697b750a614503aabae189c6f18d2eedb9f"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="f6df169"; rev="f6df1697b750a614503aabae189c6f18d2eedb9f"; } ./pkgs/t
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
@peti peti deleted the peti:t/parallel-ghc-builds branch May 7, 2020
@expipiplus1
Copy link
Contributor

expipiplus1 commented Jun 12, 2020

This change causes an uncomfortably high load on my poor machine.

image

I've also noticed loads of 120+ on my 8(16) core desktop, surely this can't be faster.

(sorry if I posted this before somewhere, I thought I did for another machine but couldn't find anything)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.