Pass -j to GHC when building packages

[kccqzy]

General summary/comments

When stack performs a build by default it does not pass -j to GHC, which causes GHC to build the modules within a package without parallelism. This is different from the -j option in stack itself, which affects how many packages can be built in parallel.

Steps to reproduce

1. Select a package with a large number of independent modules such as pandoc. Clone the repo (git clone https://github.com/jgm/pandoc.git).
2. Run command stack build within the repo.
3. Observe that when building pandoc, GHC only uses one CPU core.
4. Now remove the previous build by stack clean.
5. Run command stack build --ghc-options=-j.
6. Observe that the GHC process uses > 100% CPU and the build completes faster overall.

Expected

Stack automatically passes -j to GHC.

Actual

Stack did not.

Stack version

$ stack --version
Version 1.7.1, Git revision 681c800873816c022739ca7ed14755e85a579565 (5807 commits) x86_64 hpack-0.28.2

Method of installation

Official binary

Further remarks

It should be noted that this could also be applied when building dependencies, although the interaction between this -j and stack's own task parallelism could complicate things.

[kccqzy]

There is a passing mention of this issue in #3435 by @mgsloan. Indeed passing -j is a pretty well-known trick to speed up builds, but stack should use it by default because a newbie user doesn't necessarily know those easy ways to speed things up and will be more frustrated.

[borsboom]

I think Stack knows enough about the build plan that it could be somewhat smart about this. For example, if you have 8 CPUs and 100 dependencies to build, it would start off with -j1 for each (since it's going to build 8 packages in parallel). But let's say it's now gotten down to 4 packages currently building (each having been started with -j1) and 2 packages remaining it could then use -j2 for each of those packages (for a total of 4*1 + 2*2 = 8 threads). Once the build plan has only a single package left to build (and nothing already building), it could use -j8.

[k-bx]

I am in favor of this (as a person who recompiles a lot), but I would like to request this to be at first an optional flag that's not enabled by default. The reason is that I use --ghc-options="-j" on my packages, and I can tell that it fails quite frequently under some circumstances, unlike the regular single-thread case. Unfortunately, I never spent time to bisect this, and maybe there's a bug in GHC trac already, but for now would be safe to at least not rush with the option.

[kccqzy]

@k-bx I've never seen this fail, and I've been using this to compile everything since last year. That's including compiling dependencies using apply-ghc-options: everything. The worst I've seen is that error messages interleaved with other output. My main worry is that not making it a default will essentially make this highly useful feature hard to discover.

[k-bx]

I'm glad that it worked for you, but I think we'd better split the adoption into two steps, one being non-default so that more people would start testing it, and then make it a default eventually. I'll check when I'll have an opportunity, but I think things were getting broken especially often when I cancelled the build.

[kccqzy]

Great! Indeed I seldom if ever cancel a build; so that might be why I don't see any issues.

I think I will be producing a PR in the next few days. But I'm new to the stack code base. Is there anything special that I should take note of before contributing?

[aryairani]

I'd love this

[mpilgrem]

I've asked about GHC's -j[<n>] flag at the Haskell Community. Taking the build of Stack as an example, and comparing (PowerShell):

stack purge
Measure-Command -Expression {stack build | Out-Default}

with

stack purge
Measure-Command -Expression {stack build --ghc-options -j | Out-Default}

the latter is about 16% faster than the former on my Windows PC.