PEP 703 -- Making the Global Interpreter Lock Optional in CPython

[colesbury]

Feature or enhancement

The steering council has accepted PEP 703. This is intended as a top-level issue to keep track of integration status.

The "up for grabs" list contains issues that no one is currently working on and are ready to be implemented. If you are interested in working on one of them, please comment on the specific issue and CC me (@colesbury) on related PRs.

Up For Grabs (comment on specific issue to take)

Beta Give feedback

No tasks being tracked yet.

Linked issues

Beta Give feedback

1. --disable-gil builds are not compatible with the limited API #111506
   3.13 topic-C-API topic-free-threading type-bug +4
2. --disable-gil builds install incompatible stable ABI extensions #111870
   topic-free-threading type-crash +2
3. Avoid changing the PYMEM_DOMAIN_RAW allocator during initialization and finalization #111924
   3.13 topic-free-threading type-bug +3
4. parking_lot.c should use _PyTime_GetMonotonicClock() when available #112606
   3.13 topic-free-threading type-bug +3
   
5. critical section should avoid parking lot when re-entering top-most held critical section #114203
   topic-free-threading type-feature +2
6. More gil-disabled type thread safety #114214
   topic-free-threading type-bug +2
7. Make the specializing interpreter thread-safe in --disable-gil builds #115999
   topic-free-threading type-feature +2
8. Make pyperformance runnable from --disable-gil builds #116024
   topic-free-threading +1
   
9. Audit all built-in modules for thread safety #116738
   17 of 120
   extension-modules topic-free-threading type-feature +3
   
10. Set up tagged pointers in the evaluation stack #117139
    0 of 2
    topic-free-threading type-feature +2
    
11. _thread.lock.release() is not thread-safe in free-threaded builds #117721
    topic-free-threading type-bug +2
12. Make TSAN tests pass with the GIL disabled in free-threaded builds #117657
    4 of 31
    topic-free-threading type-feature +2
    
13. Implement the fast path for list.__getitem__ #119053
    performance topic-free-threading type-feature +3

Completed Issues

Completed Issues

Beta Give feedback

1. Add --disable-gil option to configure #108223
   3.13 build type-feature +3
2. Add --disable-gil option for Windows builds #108374
   3.13 OS-windows build type-feature +4
3. Expand pyatomic coverage #108337
   3.13 topic-C-API type-feature +3
4. Add lightweight locking C API #108724
   3.13 topic-C-API type-feature +3
   
5. Add new states ATTACHED, DETACHED, GC to PyThreadState to support PEP 703 #109549
   3.13 type-feature +2
   
6. Include "t" in ABI tag for --disable-gil builds #109740
   3.13 build type-feature +3
   
7. --disable-gil test failures due to ABI flag #110119
   3.13 tests type-bug +3
   
8. Support --disable-gil builds (PEP 703) in packaging.tags pypa/packaging#727
   packaging.tags +1
9. Add GitHub Action for checking free-threaded build #111062
   topic-free-threading +1
   
10. Implement Python Critical Sections from PEP 703 #111569
    3.13 OS-wasi topic-free-threading type-feature +4
    
11. Make _struct module thread-safe in --disable-gil builds #112062
    3.13 extension-modules topic-free-threading type-feature +4
    
12. Argument Clinic: Add support for PEP 703's critical sections (Py_BEGIN_CRITICAL_SECTION()) #111903
    3.13 topic-argument-clinic topic-free-threading type-feature +4
    
13. Make hashlib related modules thread-safe without the GIL #111916
    3.13 extension-modules topic-free-threading type-feature +4
    
14. json: make "memo" dict local to scan_once call #111928
    3.13 extension-modules topic-free-threading type-feature +4
    
15. Make functools.lru_cache thread-safe in --disable-gil builds #112070
    3.13 extension-modules topic-free-threading type-feature +4
    
16. Thread-safe one-time initialization #111956
    3.13 topic-free-threading type-feature +3
    
17. Use critical sections to protect I/O objects (in --disable-gil builds) #111965
    3.13 topic-free-threading type-feature +3
18. Support specifying object to lock in Argument Clinic's @critical_section directive #112213
    3.13 topic-argument-clinic topic-free-threading type-feature +4
    
19. Make random.Random thread-safe in --disable-gil builds #112071
    3.13 extension-modules topic-free-threading type-feature +4
    
20. unicode: make ucnhash_capi initialization thread-safe in --disable-gil builds #111972
    3.13 topic-free-threading type-feature +3
    
21. Add internal-only _PyThreadStateImpl "wrapper" for PyThreadState #112538
    3.13 interpreter-core topic-free-threading type-feature +4
22. dtoa: thread safety in --disable-gil builds #111962
    3.13 topic-free-threading type-feature +3
    
23. Make _Py_ThreadId() work on PowerPC, IBM Z, etc. #112535
    3.13 topic-free-threading type-feature +3
    
24. Support getters and setters in Argument Clinic #112205
    3.13 topic-argument-clinic topic-free-threading type-feature +4
    
25. unicode: make _PyUnicode_FromId thread-safe in --disable-gil builds #111971
    3.13 topic-free-threading type-feature +3
    
26. Py_GIL_DISABLED not defined in "free-threaded" Windows C-API extensions #111650
    3.13 topic-free-threading type-bug +3
    
27. Fix Python object resurrection on free-threaded builds #113750
    3.13 topic-free-threading type-bug +3
    
28. Implement stop-the-world functionality (for --disable-gil builds) #111964
    3.13 topic-free-threading type-feature +3
    
29. Make queue.SimpleQueue thread-safe in --disable-gil builds #113884
    topic-free-threading type-feature +2
    
30. Do not allocate non-PyObjects with PyObject_Malloc() #114569
    topic-C-API topic-free-threading type-feature +3
31. Add PyList_GetItemRef, a variant of PyList_GetItem that returns a strong reference #114329
    topic-C-API topic-free-threading type-feature +3
32. Provide a variant of PyDict_SetDefault that returns a new reference (instead of a borrowed reference) #112066
    3.13 topic-C-API topic-free-threading type-feature +4
    
33. Implement biased reference counting in --disable-gil builds #110481
    3.13 interpreter-core type-feature +3
    
34. mimalloc: additional integration and changes for --disable-gil builds #112532
    3.13 interpreter-core topic-free-threading type-feature +4
    
35. Make collections.deque thread-safe in --disable-gil builds #112050
    3.13 extension-modules topic-free-threading type-feature +4
    
36. Use per-thread freelists in --disable-gil builds #111968
    3.13 topic-free-threading type-feature +3
    
37. Keep ob_tid, ob_ref_local, and ob_ref_shared fields valid across allocations in free-threaded build #115491
    topic-free-threading type-feature +2
    
38. Move the eval_breaker to PyThreadState #112175
    3.13 interpreter-core topic-free-threading type-feature +4
    
39. Make the MRO cache thread-safe in free-threaded builds #113743
    3.13 interpreter-core topic-free-threading type-feature +4
    
40. Add delayed reclamation mechanism for free-threaded build (QSBR) #115103
    topic-free-threading type-feature +2
    
41. Make list objects thread-safe in --disable-gil builds #112087
    3.13 topic-free-threading type-feature +3
    
42. socketmodule.c: use atomics to access defaulttimeout in free-threaded build #116616
    extension-modules topic-free-threading type-feature +3
43. Add a mechanism to disable the GIL #116167
    interpreter-core topic-free-threading type-feature +3
    
44. Add support for thread sanitizer (TSAN) via --with-thread-sanitizer #112536
    3.13 build topic-free-threading type-feature +4
    
45. Use stop-the-world to make fork and interpreter shutdown thread-safe #116522
    topic-free-threading type-feature +2
    
46. Make _warnings.c thread-safe in free-threaded build #116664
    topic-free-threading type-feature +2
    
47. Make cell thread-safe in the free-threaded build #117323
    topic-free-threading type-feature +2
48. Use stop-the-world to make sys._current_frames() and sys._current_exceptions() thread-safe in free-threaded build #117300
    topic-free-threading type-feature +2
    
49. Make weakref thread-safe without the GIL #111926
    3.13 topic-free-threading type-feature +3
    
50. test.libregrtest race condition in runtest_mp leads to 30 second delay in free-threaded build #117293
    tests topic-free-threading +2
    
51. Make multiprocessing SemLock thread-safe in the free-threaded build #117435
    topic-free-threading type-feature +2
52. Make _threadmodule.c thread-safe in --disable-gil builds #114271
    topic-free-threading type-feature +2
    
53. Make set thread-safe in --disable-gil builds #112069
    3.13 topic-free-threading type-feature +3
    
54. Disable the GIL by default in free-threaded builds #116329
    interpreter-core topic-free-threading type-feature +3
    
55. Make sys.settrace, sys.setprofile, and sys.monitoring thread safe in --disable-gil builds #116818
    topic-free-threading type-feature +2
    
56. Add sys._is_gil_enabled() #117514
    topic-free-threading type-feature +2
57. Make the garbage collector thread-safe in --disable-gil builds #112529
    3.13 topic-free-threading type-feature +3
    
58. Make dict objects thread-safe in --disable-gil builds #112075
    3.13 topic-free-threading type-feature +3
    
59. Allow C extensions to declare compatibility with free-threading #116322
    interpreter-core topic-free-threading type-feature +3
    

Deferred tasks

Beta Give feedback

1. Revisit and update --disable-gil configure documentation
2. Reenable test_cppext on --disable-gil builds
3. Audit usage of PyObject_Malloc for non-PyObject allocations
4. Consider avoiding refcounting tp_mro during PyType_IsSubtype

Upstream functionality from nogil-3.12

This is a list of commits from the nogil-3.12 PR plan. The crossed-out entries are commits that do not need to be upstreamed, usually because the functionality is already in the main branch.

• cefe5dfee9 configure: disallow "--with-trace-refs" for "--disable-gil" builds
• dcddbe2ddb configure: add support for --with-thread-sanitizer (Add support for thread sanitizer (TSAN) via --with-thread-sanitizer #112536)
• f546dbf16a Enable/disable the GIL at runtime (Add a mechanism to disable the GIL #116167)
• f30d8d8f50 Add pyatomic.h
• 385eb1d99c pyport: add new macros
• de2be447b3 Make PyThreadState_GET thread-local
• a24dc2ecc3 pystate: keep track of attached vs. detached state
• 4584be5950 parking_lot: add mutexes and one-time notifications
• 6845b133cc critical_section: helpers for fine-grained locking (Implement Python Critical Sections from PEP 703 #111569)
• 8ed62cab6a pystate: use _PyRawMutex for internal mutexes (Avoid changing the PYMEM_DOMAIN_RAW allocator during initialization and finalization #111924)
• e15443b1f2 ceval: move eval_breaker to per-thread state (Move the eval_breaker to PyThreadState #112175)
• b6b12a9a94 Implement biased reference counting
• b6b12a9a94 Implement BRC inter-thread queue (Implement biased reference counting in --disable-gil builds #110481)
• 7b6b6f1a01 unicode: immortalize interned strings
• fc173e3711 unicode: always immortalize interned strings
• dd9b78460c Add safe memory reclamation scheme based on FreeBSD's GUS
• 901e134921 Add mimalloc v2.0.9 (DinoV)
• b6980856df Additional mimalloc changes (mimalloc: additional integration and changes for --disable-gil builds #112532)
• d13c63dee9 pymem: remove uses of _PyMem_SetDefaultAllocator during finalization (Avoid changing the PYMEM_DOMAIN_RAW allocator during initialization and finalization #111924)
• 654be8ffd6 gc: make the garbage collector non-generational (Make the garbage collector thread-safe in --disable-gil builds #112529)
• 967fe31473 gc: Traverese mimalloc heaps to find all objects. (Make the garbage collector thread-safe in --disable-gil builds #112529)
• 2864b6b36e Implement stop-the-world pauses (Implement stop-the-world functionality (for --disable-gil builds) #111964)
• 2864b6b36e gc: implement stop-the-world GC (Make the garbage collector thread-safe in --disable-gil builds #112529)
• c1befd7689 Stop the world before fork() and Python shutdown (Use stop-the-world to make fork and interpreter shutdown thread-safe #116522)
• 82800d8ec8 ceval: stop the world when enabling profiling/tracing for all threads (Make sys.settrace, sys.setprofile, and sys.monitoring thread safe in --disable-gil builds #116818)
• 7423dff344 pystate: use stop-the-world in a few places (Use stop-the-world to make sys._current_frames() and sys._current_exceptions() thread-safe in free-threaded build #117300)
• 86efa7dfe3 pystate: implement _PyRuntime.multithreaded
• d896dfc8db dict: make dict thread-safe (Make dict objects thread-safe in --disable-gil builds #112075)
• df4c51f82b list: make list thread-safe (Make list objects thread-safe in --disable-gil builds #112087)
• 9c1f7ba1b4 mro: thread-safe MRO cache (Make the MRO cache thread-safe in free-threaded builds #113743)
• 7a7aca096b getargs.c: make parser_init thread-safe (Thread-safe one-time initialization #111956)
• 0dddcb6f9d weakref: make weakrefs thread-safe without the GIL (Make weakref thread-safe without the GIL #111926)
• 410ba1036a dtoa: make dtoa thread-safe (dtoa: thread safety in --disable-gil builds #111962)
• 6540bf3e6a unicode: make unicodeobject.c thread-safe (unicode: make _PyUnicode_FromId thread-safe in --disable-gil builds #111971)
• 5d006db9fa codecs.c: fix race condition (unicode: make ucnhash_capi initialization thread-safe in --disable-gil builds #111972)
• d1b5ed128e _threadmodule: make _thread.lock thread-safe (Make _threadmodule.c thread-safe in --disable-gil builds #114271)
• cfc11bcb1a typeobject: thread safety
• cfecf6f4eb threading: remove _tstate_lock from threading.Thread (Make _threadmodule.c thread-safe in --disable-gil builds #114271)
• 74df7785f5 pyqueue: add internal queue data structure
• 4450445f51 pymem: add _PyMem_FreeQsbr (Add delayed reclamation mechanism for free-threaded build (QSBR) #115103)
• 7e60a01aee queue: make SimpleQueue thread-safe (Make queue.SimpleQueue thread-safe in --disable-gil builds #113884)
• 4ca2924f0d set: make set thread-safe (Make set thread-safe in --disable-gil builds #112069)
• 3cfbc49229 moduleobject: fix data races
• 5722416ef5 _threadmodule: thread-safety fixes (Make _threadmodule.c thread-safe in --disable-gil builds #114271)
• 31ec6f0290 pystate: refcount threads to handle race between interpreter shutdown and thread exit
• 45bdd27ee5 threading: make _thread.lock thread-safe
• 07f5f8c318 slice: move slice_cache to per-thread state (Use per-thread freelists in --disable-gil builds #111968)
• ea1160c6d7 asyncio: fix race conditions in enter_task and leave_task
• 212fef480e object.c: fix race when accessing attributes and methods (Use PyDict_GetItemRef() #111789)
• 70856f126d asdl: use _PyOnceFlag in Python-ast.c (Thread-safe one-time initialization #111956)
• 360a79cb88 socketmodule.c: use relaxed atomics for global 'defaulttimeout' (socketmodule.c: use atomics to access defaulttimeout in free-threaded build  #116616)
• 041a08e339 functools: make lru_cache thread-safe (Make functools.lru_cache thread-safe in --disable-gil builds #112070)
• 9bf62ffc4b random: add a mutex to guard random.Random (Make random.Random thread-safe in --disable-gil builds #112071)
• a8251a8d25 clinic: support '@' syntax for recursive mutexes (Argument Clinic: Add support for PEP 703's critical sections (Py_BEGIN_CRITICAL_SECTION()) #111903)
• ffade9d6f6 bufferedio: add locks to make print() thread-safe (Use critical sections to protect I/O objects (in --disable-gil builds) #111965)
• 5b83c16dcd textio: add locks to make textio thread-safe (Use critical sections to protect I/O objects (in --disable-gil builds) #111965)
• 6323ca60f9 stringio: make stringio thread-safe (Use critical sections to protect I/O objects (in --disable-gil builds) #111965)
• f1e4742eaa deque: make most functions thread-safe (Make collections.deque thread-safe in --disable-gil builds #112050)
• 78825e0508 importlib: fix data race in imports (PyImport_ImportModuleLevelObject)
• 2f5c90a284 semaphore.c: decrease count before release sem_lock (Make multiprocessing SemLock thread-safe in the free-threaded build #117435)
• 22eca6e215 sha1: make sha1module thread-safe (Make hashlib related modules thread-safe without the GIL #111916)
• ada9b73feb _struct: fix race condition in cache_struct_converter (Make _struct module thread-safe in --disable-gil builds #112062)
• 0e0b3899d1 signalmodule: fix thread-safety issue on macOS (Unclear if this is still an issue on macOS)
• 964bb33962 json: make JSON scanner thread safe (json: make "memo" dict local to scan_once call #111928)
• 86e7772c64 http: fix dependency on finalization order
• 9ab96964e7 faulthandler: don't dump all threads when running without the GIL
• cff32694a4 test_gdb: skip test_threads when running without GIL
• 2ae5ee5ed4 tests: fix and work around some race conditions in tests
• 9f9b3d085f ceval: fix some thread-safety issues
• 2a4c17e896 pystate: move freelists to per-thread state (Use per-thread freelists in --disable-gil builds #111968)
• 149ea9dc43 Deferred reference counting
• 7e7568672d specialize: make specialization thread-safe
• 90d34f0d18 specialize: optimize for single-threaded programs
• 42d3e11d8c code: make code object use deferred reference counting
• c9fc49666c test: add support for checking for TSAN
• 7507a77a98 thread: don't use sem_clockwait with TSAN
• a62d37674c _posixsubprocess: disable vfork when running with ASAN
• 398204d57b object: fix reported TSAN races
• 4526c07cae Disable the GIL by default in --disable-gil builds (Disable the GIL by default in free-threaded builds #116329)