Enabling CPython multi-core parallelism via subinterpreters.
This repo is for tracking the effort and as a place to keep any tooling.
From my python-ideas post (June 2015):
Python's multi-core story is murky at best. Not only can we be more clear on the matter, we can improve Python's support. The result of any effort must make multi-core (and concurrency) support in Python obvious, unmistakable, and undeniable (and keep it Pythonic).
The goal of this project is to do just that, by using the existing subinterpreter C-API.
The minimal multi-core solution will involve:
- resolving existing bugs and blockers
- exposing the existing support (from C-API) in a stdlib module
- adding a mechanism to safely "share" basic immutable objects between interpreters
- no longer sharing the GIL between interpreters
Once a minimal solution is in place we can expand from there.
At a high level, we're doing the following concrete tasks:
- PEP 554 (Multiple Interpreters in the Stdlib)
- improve interpreter isolation (especially by moving fields from
Constraints / Requirements
- no significant impact on single-threaded performance
- maintain backward compatibility (C-API, etc.)
- (pseudo-)compatibility with multiprocessing/threading/concurrent.futures APIs
- a multi-core concurrency model/approach that fits our brains
- Python APIs
- supportable on other Python implementations