Support jobserver when parallelising builds

[arnetheduck]

Summary

jobserver is a protocol for sharing parallel execution slots between parts of the compilation pipeline and nim should participate.

Description

When compiling 2 nim programs in parallel, nim will perform the builds inefficiently by overscheduling jobs causing build fails due to excessive memory usage or cause slow compile times due to heavy context switching and sub-optimal scheduling (in a build where there are 4 cpu:s and one nim process is compiling C files, the other nim process could start compiling nim code as soon as there are 3 c compilations ongoing, instead of waiting for all c steps to finish)

When using a modern toolchain with LTO, the problem is exacerbated because either LTO gets launched without multi-processing support which causes it to compile extremely slowly or it uses available parallelism again contributing to overscheduling of jobs.

For this reason, gcc, make etc support managing parallelism across several tools and so should nim (specially when it detects that it's running inside a jobserver-based environment like make) - this is simple to implement (example minimal rust code) and would allow managing nim resources more sanely while utilizing available parallelism optimally.

The slot management rules are simple:

• while nim is compiling nim code it should borrow a slot and give it back when done
• for every c file that it compiles, borrow a slot
• linking? hold a slot

Ideally, nim would also re-launch itself for the compile-to-c step as nim tends to hold on to memory while compiling C file - on constrained devices, this sometimes causes avoidable memory exhaustion.

In nimbus-build-system we manage this by letting make drive the build, compiling nim separately and letting nim generate command lines for the C code it generates, then we compile the c code with make and link with make as well. While it optimally uses the -j job slots given to it and thus significantly speeds up builds, it's a messy toolchain to maintain over time - it would be a lot better if nim just did this automatically out-of-the-box.

Alternatives

No response

Examples

No response

Backwards Compatibility

No response

Links

No response