Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

compileall with workers would is faster with larger chunksize #117205

Closed
hauntsaninja opened this issue Mar 25, 2024 · 1 comment
Closed

compileall with workers would is faster with larger chunksize #117205

hauntsaninja opened this issue Mar 25, 2024 · 1 comment
Labels
performance Performance or resource usage stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@hauntsaninja
Copy link
Contributor

hauntsaninja commented Mar 25, 2024 •
edited by bedevere-app bot

Feature or enhancement

Proposal:

Currently we do a map with the default chunksize=1 over files:

cpython/Lib/compileall.py

Line 109 in 3726cb0

results = executor.map(partial(compile_file,

A quick local benchmark suggests that chunksize=4 would be 20% faster

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs
@hauntsaninja hauntsaninja added the type-feature A feature request or enhancement label Mar 25, 2024
@bedevere-app bedevere-app bot mentioned this issue Mar 25, 2024
Merged
@Eclips4 Eclips4 added performance Performance or resource usage stdlib Python modules in the Lib dir labels Mar 25, 2024
hauntsaninja added a commit that referenced this issue Apr 3, 2024
@hauntsaninja
gh-117205: Increase chunksize when compiling pyc in parallel (#117206)
b4fe02f
@JelleZijlstra
Copy link
Member

Looks like this has been done, thanks!

diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@hauntsaninja @diegorusso
pythongh-117205: Increase chunksize when compiling pyc in parallel (p…
aef97cb 
…ython#117206)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance Performance or resource usage stdlib Python modules in the Lib dir type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@JelleZijlstra @hauntsaninja @Eclips4