Deadlock caused by type_lock_prevent_release()

[nascheme]

Bug report

Bug description:

The current implementation of type_lock_prevent_release() has a flaw. If the critical section holds both TYPE_LOCK and the type dict mutex (this happens with BEGIN_TYPE_DICT_LOCK), then we can end up deadlocking. This is because only the type lock is prevented from being released and the dict mutex is released. This results in a lock inversion.

The suggested fix is to prevent release of both held by the top critical section.

Reproducer (note it depends on the memory addresses of the mutexes so not 100% reliable).

def test_concurrent_setattr_deadlock():
    # two threads assigning to a special method of the same
    # class could deadlock.  One thread held the type lock and waited for
    # the type dict mutex, which its critical section had released when it
    # blocked on the stop-the-world mutex, while the other held the type
    # dict mutex and waited for the type lock.
    class Base:
        pass

    N = 2000
    done = False

    def setter():
        func = lambda self: "x"
        while not done:
            Base.__repr__ = func
            try:
                del Base.__repr__
            except AttributeError:
                pass

    def subclasser():
        while not done:
            type('Sub', (Base,), {})()

    def lister():
        while not done:
            Base.__subclasses__()

    def basesetter():
        nonlocal done
        for i in range(N):
            if i % 100 == 0:
                print(i)

            class A:
                pass

            class C:
                pass

            class B(A):
                pass

            B.__bases__ = (C,)
        done = True

    # The setter threads are the ones that deadlock.  The others are there
    # to keep the type lock and the stop-the-world mutex contended, which
    # is what gets the setters into the window where it happens.
    targets = (
        setter,
        setter,
        subclasser,
        subclasser,
        lister,
        lister,
        basesetter,
    )
    threads = [Thread(target=target) for target in targets]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-155400: Fix deadlock with type_lock_prevent_release() #155401