Make the Python test suite pass with the GIL disabled

[colesbury]

Feature or enhancement

This issue is to track the remaining bugs that prevent the Python test suite from passing with the GIL disabled, as well as the PRs under review that fix known issues.

Note that this is not intended to cover all free-threading related issues: some thread-safety issues don't result in tests crashing or failing and are not tracked here. This is intended to cover the minimum set of changes needed to ensure that the test suite passes reliably with the GIL disabled.

PRs

• gh-112075: _Py_dict_lookup needs to lock shared keys #117528
• gh-112075: Lock when inserting into shared keys #117824
• gh-117783: Immortalize objects that use deferred reference counting #118112

Integration branch

• https://github.com/colesbury/cpython/tree/nogil-integration

Branches to merge

git merge disable-gil-default &&
git merge gh-112075-lock-split-keys &&
git merge gh-117783-immortalize

This branch contains the merge of the above PRs and is used to find bugs that are not covered by outstanding PRs. Note that setting PYTHON_GIL=0 isn't sufficient because some tests do not propagate the environment variable to tests in subprocesses. Instead, the branch changes the default mode to disable the GIL for testing purposes.

Issues without PRs that fix them

• N/A

Bugs without issues

Windows

• test_no_stale_references times out - https://github.com/python/cpython/actions/runs/8514458340/job/23320231100?pr=117437

Linux

• test.test_concurrent_futures.test_deadlock deadlocks ? - https://buildbot.python.org/all/#/builders/1281/builds/148/steps/5/logs/stdio

macOS

• test_interrupt_main_subthread - RuntimeError: release unlocked lock https://buildbot.python.org/all/#/builders/1367/builds/117/steps/5/logs/stdio
• test.test_multiprocessing_spawn.test_processes.WithProcessesTestPool.test_release_task_refs - 10 != 0

https://buildbot.python.org/all/#/builders/1367/builds/119/steps/5/logs/stdio

• test_daemon_threads_shutdown_stderr_deadlock - Assertion failed: (tstate == NULL || tstate->interp == interp), function _PyEval_ReleaseLock, file ceval_gil.c,
• test_queue test_shutdown_all_methods_in_many_threads - AttributeError: 'Queue' object has no attribute '_lock'

test_shutdown_all_methods_in_many_threads (test.test_queue.CQueueTest.test_shutdown_all_methods_in_many_threads) ... Warning -- Uncaught thread exception: AttributeError
Exception in thread Thread-2081 (_read_msg_thread):
Traceback (most recent call last):
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/threading.py", line 1039, in _bootstrap_inner
    self.run()
    ~~~~~~~~^^
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/threading.py", line 990, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/test_queue.py", line 352, in _read_msg_thread
    q.get(False)
    ~~~~~^^^^^^^
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/queue.py", line 216, in get
    item = self._get()
           ~~~~~~~~~^^
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/threading.py", line 304, in __enter__
    return self._lock.__enter__()
           ^^^^^^^^^^
AttributeError: 'Queue' object has no attribute '_lock'

Old bugs?

• test.test_smtpnet.SmtpSSLTest.test_connect_default_port connection closed (possibly related)
• test.test_multiprocessing_forkserver.test_processes hangs ?

Linked PRs

• gh-116749: Run test suite with the GIL disabled #117437
• gh-116749: Disable GIL by default in free-threaded build #118295

[tonybaloney]

Partly related: #112677 which tests inside a sub interpreter (with each test having an isolated GIL).

[colesbury]

The relevant PRs are merged, so I'm closing this now.

We'll open new issues for bugs/crashes as they pop up.