distutils doesn't parallelize extension module compilation

[pitrou]

BPO	5309
Nosy	@pitrou, @scoder, @vstinner, @giampaolo, @tiran, @tarekziade, @merwok, @berkerpeksag
Files	build_ext_parallel.patch build_ext_parallel2.patch build_ext_parallel3.patch build_ext_parallel4.patch build_ext_parallel5.patch build_ext_parallel6.patch issue5309-doc.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-09-30.13:04:24.612>
created_at = <Date 2009-02-19.01:39:51.797>
labels = ['type-feature', 'build']
title = "distutils doesn't parallelize extension module compilation"
updated_at = <Date 2016-09-17.19:05:32.783>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2016-09-17.19:05:32.783>
actor = 'giampaolo.rodola'
assignee = 'none'
closed = True
closed_date = <Date 2014-09-30.13:04:24.612>
closer = 'pitrou'
components = ['Build']
creation = <Date 2009-02-19.01:39:51.797>
creator = 'pitrou'
dependencies = []
files = ['32378', '32380', '32382', '36545', '36564', '36671', '37824']
hgrepos = []
issue_num = 5309
keywords = ['patch', 'gsoc']
message_count = 30.0
messages = ['82447', '116603', '178993', '179010', '201392', '201396', '201400', '226412', '226415', '226424', '226521', '226523', '226546', '226646', '227188', '227651', '227652', '227877', '227883', '227885', '227887', '227888', '227902', '227906', '227907', '234147', '234514', '234803', '240505', '276819']
nosy_count = 13.0
nosy_names = ['pitrou', 'scoder', 'vstinner', 'giampaolo.rodola', 'christian.heimes', 'tarek', 'eric.araujo', 'Arfrever', 'python-dev', 'jtaylor', 'berker.peksag', 'bfroehle', 'Sjlver']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue5309'
versions = ['Python 3.5']

[pitrou]

When run with "make -jN", the Python compilation process is able to
parallelize across N concurrent processes for the interpreter core, but
extension modules are still compiled sequentially.

[merwok]

Do you want to work on a patch?

[pitrou]

The original request is really about setup.py, not packaging. I don't care about packaging, and it's not in the stdlib.

[tiran]

+1

I think it's sufficient to parallelize the compilation step (.c -> .o) and ignore the linker step. Linking is usually so fast that it doesn't matter.

Idea:

• separate compile step from link step
• run all compile calls for all extensions in parallel until all objects of all extensions are generated
• do linking and error reporting in serial mode

[pitrou]

Here is a working patch.

[pitrou]

Here is an updated patch which only enables parallel building in setup.py.