Support parallel builds when installing with pip?

[jacklovell]

Installing raysect from source using pip takes a long time (15 minutes in our Cherab CI for example), largely because the compilation of all the Cython extension modules is done in serial. This could be sped up by parallelising the build process. This is already done for the cythonize step (transpiling pyx to c), using all available cores. Doing it for the build step is pretty trivial: add the following in the setup call in setup.py:

setup(
    name="raysect",
    ...
    options={'build_ext': {'parallel': multiprocessing.cpu_count()}},
)

Or, for more explicit parallelisation, an environment variable to specify the number of parallel processes, defaulting to serial:

ncpu = os.getenv('RAYSECT_NCPU', '1')
setup(
    name="raysect",
    ...
    options={'build_ext': {'parallel': ncpu}},
)

N.B. Command line arguments to setup.py don't work well when installing via pip: passing --global-option means pip won't use wheels for any dependencies (Cython, numpy and matplotlib in particular cause problems) which is why an environment variable is preferred.

[CnlPepper]

Have they finally added support for parallel compilation? Excellent I'll add that for the next release. Thanks for finding this I, barely have time to look at packaging etc.. anymore.

[jacklovell]

I'm pretty sure it's always been there (this is a distutils feature), but poorly documented. Difference is, in the past you'd download and extract the source tarball and run python setup.py build_ext --parallel={ncpu} for the parallel build, whereas in the brave new world of PEP517 it's deprecated to call setup.py directly and this was recommended as the alternative in a discourse.python.org thread I found.

[CnlPepper]

I was aware that distutils supported parallel compilation (we use it in dev/build.sh), but I never found a reliable method to get it to work with pip when I last looked (years ago!). Of the options above, I think I'd just force it to always use all cores. I'll make it override-able with an environment variable.

[jacklovell]

Fixed in #428