Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

simple parallel build #52

Closed
wants to merge 1 commit into from
Closed

simple parallel build #52

wants to merge 1 commit into from

Conversation

dimaqq
Copy link

@dimaqq dimaqq commented Feb 25, 2021

Compile .c.o in parallel to close #28

Tries to use as many cores as available, naively, but already better than one file at a time:

Screenshot 2021-02-25 at 12 56 26

@adrianeboyd
Copy link
Collaborator

If we do this (I still need to look at the details), instead of the max cpu count we should use the -j option which is stored as self.parallel in ExtensionBuilder.

@dimaqq
Copy link
Author

dimaqq commented Feb 26, 2021

re: configurability
I don't mind at all if you take over this PR :)

re: automated checks...
I see that Python 2.7 and 3.5 checks are still configured for this project, however, the only wheels are only available for 3.6 ~ 3.9 on PYPI.
I'm not sure what Python versions should ultimately be supported.
I think that f-string is the only "new" feature in the proposed change.

@adrianeboyd
Copy link
Collaborator

We have a wider range of python version compatibility than published wheels in a lot of cases, for instance we've kept spacy v2.x compatible with python 2.7 even though we haven't published python 2.7 wheels for a while.

I have to look a bit more into the details about what kind of multiprocessing setup makes sense, since I'd rather have slow builds than broken/brittle builds.

And probably it does make sense to do this in combination with an environment variable so this also works with pip and other build systems, something like numpy's NPY_NUM_BUILD_JOBS.

@dimaqq
Copy link
Author

dimaqq commented Jan 25, 2022

This has probably gotten stale, feel free to close.

@svlandeg svlandeg closed this Apr 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

support parallel builds with setup.py build -j10
3 participants