Compileall script: add option to use multiple cores

[dholth]

BPO	16104
Nosy	@brettcannon, @gpshead, @merwok, @dholth, @PCManticore, @JimJJewett
Files	compileall_v1.patch issue16104.patch: Remove trailing whitespaces issue16104_1.patch issue16104_2.patch: Minor doc fixes. issue16104_3.patch issue16104_4.patch issue16104_5.patch issue16104_6.patch issue16104_7.patch: Skip a couple of tests if concurrent.futures is unavailable issue16104_8.patch issue16104_9.patch issue16104_10.patch issue16104_11.patch: Minor doc update issue16104_12.patch 16104.patch: Update the patch for the tip.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2014-09-12.14:40:30.609>
created_at = <Date 2012-10-01.20:46:54.073>
labels = ['type-feature', 'library']
title = 'Compileall script: add option to use multiple cores'
updated_at = <Date 2020-02-27.08:02:38.705>
user = 'https://github.com/dholth'

bugs.python.org fields:

activity = <Date 2020-02-27.08:02:38.705>
actor = 'gregory.p.smith'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2014-09-12.14:40:30.609>
closer = 'brett.cannon'
components = ['Library (Lib)']
creation = <Date 2012-10-01.20:46:54.073>
creator = 'dholth'
dependencies = []
files = ['33079', '34339', '34368', '34381', '34383', '34384', '34400', '34401', '34404', '35018', '35054', '35055', '35056', '35106', '36590']
hgrepos = []
issue_num = 16104
keywords = ['patch']
message_count = 28.0
messages = ['171744', '171758', '205805', '213200', '213209', '213298', '213301', '213303', '213304', '213307', '213308', '213317', '213340', '213417', '213419', '213422', '214450', '217118', '217173', '217261', '217264', '217399', '217586', '226684', '226822', '226823', '226824', '362786']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'gregory.p.smith', 'eric.araujo', 'dholth', 'Claudiu.Popa', 'python-dev', 'Jim.Jewett']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue16104'
versions = ['Python 3.5']

[dholth]

compileall would benefit approximately linearly from additional CPU cores. There should be an option.

The noisy output would have to change. Right now it prints "compiling" and then "done" synchronously with doing the actual work.

[brettcannon]

This should probably use concurrent.futures instead of multiprocessing directly, but yes it would be useful.

Then again, the whole module should probably be rewritten to use importlib as well.

[PCManticore]

Hello!

Here's a draft patch. It adds a new *processes* parameter to *compile_dir* and a new command line parameter as well.

[merwok]

Patch looks good. Some comments on Rietveld.

[PCManticore]

Thank you for the review, Éric! Here's the updated patch.

[merwok]

FTR, py_compile and compileall use importlib in 3.4.

[merwok]

This looks ready to me.

One thing: “make -j0” is the spelling for “run using all available cores”, whereas “compileall -j0” will use one process. I don’t know if this should be documented, changed or ignored.

[brettcannon]

I vote for changed so that -j0 uses all available cores as os.cpu_count() states.

[PCManticore]

I agree. I'll modify the patch.

[merwok]

+ if args.processes <= 0:
Is that correct? For make, I think I’ve always seen “-j0”, not negative values.

Could you add a test for -j0? (i.e. check that “compileall -j0” calls the function with “processes=os.cpu_count()”)

[PCManticore]

regrtest does that, checking for j <=0.