gh-128679: Fix tracemalloc.stop() race conditions

[vstinner]

tracemalloc_alloc(), tracemalloc_realloc(), tracemalloc_free(), _PyTraceMalloc_TraceRef() and _PyTraceMalloc_GetMemory() now check tracemalloc_config.tracing after calling TABLES_LOCK().

_PyTraceMalloc_TraceRef() now always returns 0.

• Issue: Race condition in tracemalloc causes segfaults #128679

[vstinner]

@ZeroIntensity: Ok, here is the new fix for #128679 tracemalloc.stop() race condition.

Thanks to #128888 refactoring, the fix is simpler and should be more reliable than my previous attempt. Moreover, this fix is more complete: I fixed more functions.

[vstinner]

For the test, I prefer to use PyThread locks since it's a public API, whereas _Py_LOCK_DONT_DETACH flag of PyMutex is only part of the internal C API.

[ZeroIntensity]

Ok, that makes sense. We might want to expose that publicly at some point, FWIW.

[ZeroIntensity]

To be perfectly clear--what are the rules for tracemalloc_config.tracing now? Is it now always protected by the tables lock, regardless of the GIL?

[vstinner]

To be perfectly clear--what are the rules for tracemalloc_config.tracing now? Is it now always protected by the tables lock, regardless of the GIL?

It's protected by TABLES_LOCK(). Basically, the whole tracemalloc state is protected by TABLES_LOCK().

The GIL is needed on memory allocations to get the traceback. It's not needed on deallocations.

[ZeroIntensity]

Thanks!

[vstinner]

Merged. Thanks for your reviews @ZeroIntensity!

[encukou]

Combined with #137994, TABLES_LOCK can cause a deadlock; see: #139116