cmd/compile: Support make jobserver for dynamic negotiable concurreny during compilation

[kunaltyagi]

What version of Go are you using (go version)?

$ go version
1.12.2 gccgo

Does this issue reproduce with the latest release?

Not tested

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/builder/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/builder/go"
GOPROXY=""
GORACE=""
GOROOT="/usr"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/gcc/x86_64-linux-gnu/9"
GCCGO="/usr/bin/x86_64-linux-gnu-gccgo-9"
CC="/opt/miniconda3/envs/builder/bin/x86_64-conda-linux-gnu-cc"
CXX="/opt/miniconda3/envs/builder/bin/x86_64-conda-linux-gnu-c++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build646202173=/tmp/go-build -gno-record-gcc-switches -funwind-tables"

What did you do?

• Clone five independent packages
• Compile each with say 2 threads
• Observer number of threads
• It goes down from 10 (max possible) to 5 (minimum)

What did you expect to see?

• Using a single workstation as CI server allows orchestration of concurrency during compilation of multiple independent packages
• Compiling go with other languages plays well in regards to optimal utilization of CPU cores

What did you see instead?

• Controlling max threads in a single go compile process is possible
• Sharing of threads between independent go compile process is not possible
• Ensuring that 5 or so independent compile processes are keeping 8 CPU's fully stocked is not feasible.

Possible solution

Implement support for GNU make jobserver for compilation. This doesn't have to be exposed for 80-90% of users whose primary use-case doesn't involve parallel independent compilation of go packages.

Bonus: Both compilation and testing can benefit from this

Pro:

• Works with make, ninja-cmake and other related build and meta build systems
• Works cross-platform on systems with sockets (all modern systems targeted by go as compilation platforms based on expectation of pulling dependencies automatically)
• Parity with compilation process of compilers like C/C++ (clang, gcc), rust (cargo)
• Allows power users to perform quick context switches during compilation and testing
• Allows people working on polyglot code bases a significant boost (a lot of tech stacks are not pure go)

Con:

• jobserver has drawbacks (the pipe could have been closed, env variables need to be passed into sub-shells (not a concern here, but often a source of errors))
• more complicated process for spawning sub-processes
• maybe low utilization in go community

[mdempsky]

Concretely and technically, what does it mean to support the "make jobserver"? Is there a spec or something that explains what this means?

Also, my impression is most Go users use "go build" or Bazel. Will addressing this issue help those users? It sounds like it would only help Go users that use GNU make, which I didn't think there were that many.

[kunaltyagi]

Make Jobserver is more concretely defined in:

• https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html
• https://www.gnu.org/software/make/manual/html_node/Job-Slots.html

Sadly, I don't think there is a "spec", but this is unlikely to change in a non-backward compatible way since GNU based on historic data.

(There are some subtleties involved, but) Essentially

1. CLI support: Read --jobserver-auth=R,W from MAKEFLAGS environment variable
2. During the build/test stage:
   1. Read N character (or more) from fd to get a "token"/permission to utilize N compute resources (aka CPU core)
   2. Create N processes (ideally. You can create more or less processes, if required. Eg: make creates N+1 processes, with the extra process for the orchestration master process)
   3. Write back the same character as soon as a sub-process completes, and goto 1 again

Does this only support users using GNU make

Please see foreign jobserver support for bazel for more details how this would help even a wider audience who are already using this for their mixed language bazel setup.
Please note that the third party script is needed not just due to issue still being open, but also to orchestrate multiple bazel builds together

In a wider context, as stated in the original comment, this would essentially allow much better utilization of the CI for pure go build style builds where the different builds don't explicitly know about each other (and are most likely running inside dockers/jails for isolation).

As a fringe benefit, this would also allow go build to play nice with meta build systems (eg: catkin and colcon, primarily used in robotics community) which target distributed and polyglot code bases.

Sadly, I don't know how the workflows of gophers look like (I'd be a very poor judge), so can't comment more generically.

EDIT:
Long-form post by Paul Smith explaining the design rationale for the interested