Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement PEP 684: A Per-Interpreter GIL #99113

Closed
ericsnowcurrently opened this issue Nov 5, 2022 · 3 comments
Closed

Implement PEP 684: A Per-Interpreter GIL #99113

ericsnowcurrently opened this issue Nov 5, 2022 · 3 comments
Labels
3.12 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters type-feature A feature request or enhancement

Comments

@ericsnowcurrently
Copy link
Member

ericsnowcurrently commented Nov 5, 2022 •
edited by bedevere-bot

While PEP 684 is not accepted yet (and could get rejected, of course), it would be helpful to have an issue against which I can start creating draft PRs for the implementation.

This issue isn't for all the prerequisite work for a per-interpreter GIL, rather just the implementation of what's actually specified in the PEP. The rest is happening separately.

(Also see my checklist: https://github.com/ericsnowcurrently/multi-core-python/wiki/0-The-Plan)
@ericsnowcurrently ericsnowcurrently added type-feature A feature request or enhancement interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters 3.12 bugs and security fixes labels Nov 5, 2022
@bedevere-bot bedevere-bot mentioned this issue Nov 5, 2022
Closed
3 tasks
This was referenced May 5, 2023
Merged
Merged
Merged
Merged
Merged
Closed
Merged
ericsnowcurrently added a commit that referenced this issue May 5, 2023
@ericsnowcurrently
gh-99113: Share the GIL via PyInterpreterState.ceval.gil (gh-104203)
55671fe 
In preparation for a per-interpreter GIL, we add PyInterpreterState.ceval.gil, set it to the shared GIL for each interpreter, and use that rather than using _PyRuntime.ceval.gil directly.  Note that _PyRuntime.ceval.gil is still the actual GIL.
@ericsnowcurrently
Copy link
Member Author

Docs updates:

ericsnowcurrently added a commit that referenced this issue May 5, 2023
@ericsnowcurrently
gh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (gh-104205)
a9c6e06 
Here we are doing no more than adding the value for Py_mod_multiple_interpreters and using it for stdlib modules.  We will start checking for it in gh-104206 (once PyInterpreterState.ceval.own_gil is added in gh-104204).
ericsnowcurrently added a commit that referenced this issue May 5, 2023
@ericsnowcurrently
gh-99113: Add PyInterpreterConfig.own_gil (gh-104204)
f3e7eb4 
We also add PyInterpreterState.ceval.own_gil to record if the interpreter actually has its own GIL.

Note that for now we don't actually respect own_gil; all interpreters still share the one GIL.  However, PyInterpreterState.ceval.own_gil does reflect PyInterpreterConfig.own_gil.  That lie is a temporary one that we will fix when the GIL really becomes per-interpreter.
carljm added a commit to carljm/cpython that referenced this issue May 5, 2023
@carljm
Merge branch 'main' into inlinecomp2
baacf5f 
* main:
  pythongh-99113: Add PyInterpreterConfig.own_gil (pythongh-104204)
  pythongh-104146: Remove unused var 'parser_body_declarations' from clinic.py (python#104214)
  pythongh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (pythongh-104205)
  pythongh-104108: Add the Py_mod_multiple_interpreters Module Def Slot (pythongh-104148)
  pythongh-99113: Share the GIL via PyInterpreterState.ceval.gil (pythongh-104203)
  pythonGH-100479: Add `pathlib.PurePath.with_segments()` (pythonGH-103975)
  pythongh-69152: Add _proxy_response_headers attribute to HTTPConnection (python#26152)
  pythongh-103533: Use PEP 669 APIs for cprofile (pythonGH-103534)
  pythonGH-96803: Add three C-API functions to make _PyInterpreterFrame less opaque for users of PEP 523. (pythonGH-96849)
ericsnowcurrently added a commit that referenced this issue May 6, 2023
@ericsnowcurrently
gh-99113: Add a check for Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (gh-10…
fff193b 
…4206)

Py_MOD_PER_INTERPRETER_GIL_SUPPORTED is a new supported value for Py_mod_multiple_interpreters, added in gh-104205.
ericsnowcurrently added a commit that referenced this issue May 6, 2023
@ericsnowcurrently
gh-99113: Make Sure the GIL is Acquired at the Right Places (gh-104208)
92d8bff 
This is a pre-requisite for a per-interpreter GIL.  Without it this change isn't strictly necessary.  However, there is no real downside otherwise.
jbower-fb pushed a commit to jbower-fb/cpython-jbowerfb that referenced this issue May 8, 2023
@ericsnowcurrently @jbower-fb
pythongh-99113: Share the GIL via PyInterpreterState.ceval.gil (pytho…
0826f80 
…ngh-104203)

In preparation for a per-interpreter GIL, we add PyInterpreterState.ceval.gil, set it to the shared GIL for each interpreter, and use that rather than using _PyRuntime.ceval.gil directly.  Note that _PyRuntime.ceval.gil is still the actual GIL.
jbower-fb pushed a commit to jbower-fb/cpython-jbowerfb that referenced this issue May 8, 2023
@ericsnowcurrently @jbower-fb
pythongh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (pythongh-10…
7192a46 
…4205)

Here we are doing no more than adding the value for Py_mod_multiple_interpreters and using it for stdlib modules.  We will start checking for it in pythongh-104206 (once PyInterpreterState.ceval.own_gil is added in pythongh-104204).
jbower-fb pushed a commit to jbower-fb/cpython-jbowerfb that referenced this issue May 8, 2023
@ericsnowcurrently @jbower-fb
pythongh-99113: Add PyInterpreterConfig.own_gil (pythongh-104204)
b057233 
We also add PyInterpreterState.ceval.own_gil to record if the interpreter actually has its own GIL.

Note that for now we don't actually respect own_gil; all interpreters still share the one GIL.  However, PyInterpreterState.ceval.own_gil does reflect PyInterpreterConfig.own_gil.  That lie is a temporary one that we will fix when the GIL really becomes per-interpreter.
jbower-fb pushed a commit to jbower-fb/cpython-jbowerfb that referenced this issue May 8, 2023
@ericsnowcurrently @jbower-fb
pythongh-99113: Add a check for Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (p…
fc1e63f 
…ythongh-104206)

Py_MOD_PER_INTERPRETER_GIL_SUPPORTED is a new supported value for Py_mod_multiple_interpreters, added in pythongh-104205.
jbower-fb pushed a commit to jbower-fb/cpython-jbowerfb that referenced this issue May 8, 2023
@ericsnowcurrently @jbower-fb
pythongh-99113: Make Sure the GIL is Acquired at the Right Places (py…
74d38ea 
…thongh-104208)

This is a pre-requisite for a per-interpreter GIL.  Without it this change isn't strictly necessary.  However, there is no real downside otherwise.
ericsnowcurrently added a commit that referenced this issue May 8, 2023
@ericsnowcurrently
gh-99113: A Per-Interpreter GIL! (gh-104210)
5c9ee49 
This is the culmination of PEP 684 (and of my 8-year long multi-core Python project)!

Each subinterpreter may now be created with its own GIL (via Py_NewInterpreterFromConfig()).  If not so configured then the interpreter will share with the main interpreter--the status quo since subinterpreters were added decades ago.  The main interpreter always has its own GIL and subinterpreters from Py_NewInterpreter() will always share with the main interpreter.
carljm added a commit to carljm/cpython that referenced this issue May 9, 2023
@carljm
Merge branch 'main' into inlinecomp2
76077cd 
* main: (47 commits)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)
  pythongh-103193: Improve `getattr_static` test coverage (python#104286)
  Trim trailing whitespace and test on CI (python#104275)
  pythongh-102500: Remove mention of bytes shorthand (python#104281)
  pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)
  pythongh-99108: Replace SHA3 implementation HACL* version (python#103597)
  pythongh-104273: Remove redundant len() calls in argparse function (python#104274)
  pythongh-64660: Don't hardcode Argument Clinic return converter result variable name (python#104200)
  pythongh-104265 Disallow instantiation of `_csv.Reader` and `_csv.Writer` (python#104266)
  pythonGH-102613: Improve performance of `pathlib.Path.rglob()` (pythonGH-104244)
  pythongh-103650: Fix perf maps address format (python#103651)
  pythonGH-89812: Churn `pathlib.Path` methods (pythonGH-104243)
  ...
carljm added a commit to carljm/cpython that referenced this issue May 9, 2023
@carljm
Merge branch 'main' into fixsendfamily
a614216 
* main: (29 commits)
  pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)
  pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)
  pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)
  require-pr-label.yml: Add missing "permissions:" (python#104309)
  pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)
  pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)
  pythongh-103193: Improve `getattr_static` test coverage (python#104286)
  Trim trailing whitespace and test on CI (python#104275)
  pythongh-102500: Remove mention of bytes shorthand (python#104281)
  pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)
  pythongh-99108: Replace SHA3 implementation HACL* version (python#103597)
  ...
carljm added a commit to carljm/cpython that referenced this issue May 9, 2023
@carljm
Merge branch 'main' into supermon
c48eee3 
* main: (156 commits)
  pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304)
  pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441)
  pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096)
  pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217)
  pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312)
  pythongh-104240: return code unit metadata from codegen (python#104300)
  pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)
  pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)
  pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)
  require-pr-label.yml: Add missing "permissions:" (python#104309)
  pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)
  pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  ...
carljm added a commit to carljm/cpython that referenced this issue May 9, 2023
@carljm
Merge branch 'main' into fixloadsuperspec
cf11908 
* main: (35 commits)
  pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304)
  pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441)
  pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096)
  pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217)
  pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312)
  pythongh-104240: return code unit metadata from codegen (python#104300)
  pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)
  pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)
  pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)
  require-pr-label.yml: Add missing "permissions:" (python#104309)
  pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)
  pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  ...
@ericsnowcurrently ericsnowcurrently mentioned this issue Jul 26, 2023
Closed
This was referenced Jul 26, 2023
Closed
Closed
@bedevere-bot bedevere-bot mentioned this issue Jul 31, 2023
Merged
ericsnowcurrently added a commit that referenced this issue Jul 31, 2023
@ericsnowcurrently
gh-99113: Add a What's New Entry for PEP 684 (gh-107520)
79e479c
@bedevere-bot bedevere-bot mentioned this issue Jul 31, 2023
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 31, 2023
@ericsnowcurrently @miss-islington
pythongh-99113: Add a What's New Entry for PEP 684 (pythongh-107520)
d68c535 
(cherry picked from commit 79e479c)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
ericsnowcurrently added a commit that referenced this issue Jul 31, 2023
@miss-islington @ericsnowcurrently
[3.12] gh-99113: Add a What's New Entry for PEP 684 (gh-107520) (gh-1…
31cd12a 
…07523)

gh-99113: Add a What's New Entry for PEP 684 (gh-107520)
(cherry picked from commit 79e479c)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
@erlend-aasland
Copy link
Contributor

@ericsnowcurrently, do you still want to keep this open? IIRC, most of the follow-up work is tracked in separate issues.

@ericsnowcurrently
Copy link
Member Author

Yeah, I think you're right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.12 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters type-feature A feature request or enhancement
Projects
Subinterpreters
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ericsnowcurrently @erlend-aasland