Leak of locks from multiprocessing.Process

[bstaletic]

BPO	43145
Nosy	@pitrou, @applio, @shihai1991, @YannickJadoul, @bstaletic
Files	log: valgrind output valgrind output.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-02-06.13:02:37.289>
labels = ['expert-C-API', '3.11', 'performance']
title = 'Leak of locks from multiprocessing.Process'
updated_at = <Date 2021-07-24.07:52:25.721>
user = 'https://github.com/bstaletic'

bugs.python.org fields:

activity = <Date 2021-07-24.07:52:25.721>
actor = 'shihai1991'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['C API']
creation = <Date 2021-02-06.13:02:37.289>
creator = 'bstaletic'
dependencies = []
files = ['49793', '50177']
hgrepos = []
issue_num = 43145
keywords = []
message_count = 4.0
messages = ['386558', '386559', '386774', '398121']
nosy_count = 5.0
nosy_names = ['pitrou', 'davin', 'shihai1991', 'YannickJadoul', 'bstaletic']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue43145'
versions = ['Python 3.11']

[bstaletic]

The following C code leaks 7 locks allocated with PyThread_allocate_lock:

#include <Python.h>

int main()
{
	Py_Initialize();
	PyObject* multiprocessing = PyImport_ImportModule("multiprocessing");
	PyObject* Process = PyObject_GetAttrString(multiprocessing, "Process");
	PyObject* args = PyTuple_New(0);
	PyObject* kw = PyDict_New();
	PyDict_SetItemString(kw, "target", Process);

	PyObject* p = PyObject_Call(Process, args, kw);
	PyObject* start = PyObject_GetAttrString(p, "start");
	PyObject* join = PyObject_GetAttrString(p, "join");

	PyObject_CallNoArgs(start);
	PyObject_CallNoArgs(join);

	Py_DECREF(join);
	Py_DECREF(start);
	Py_DECREF(p);
	Py_DECREF(kw);
	Py_DECREF(args);
	Py_DECREF(Process);
	Py_DECREF(multiprocessing);
	Py_Finalize();
}

The following locks are leaked:

1. cpython/Python/pystate.c

   Line 78 in 196d4de

   runtime->interpreters.mutex = PyThread_allocate_lock();

2. cpython/Python/pystate.c

   Line 84 in 196d4de

   runtime->xidregistry.mutex = PyThread_allocate_lock();

3. cpython/Python/pystate.c

   Line 90 in 196d4de

   runtime->main_thread = PyThread_get_thread_ident();

4. https://github.com/python/cpython/blob/master/Python/ceval.c#L810
5. https://github.com/python/cpython/blob/master/Python/import.c#L126
6. and 7. https://github.com/python/cpython/blob/master/Modules/_threadmodule.c#L597

In the attachment is valgrind's output.

[bstaletic]

Slightly simpler C example:

#include <Python.h>

int main()
{
	Py_Initialize();
	PyObject* multiprocessing = PyImport_ImportModule("multiprocessing");
	PyObject* Process = PyObject_GetAttrString(multiprocessing, "Process");

	PyObject* p = PyObject_CallNoArgs(Process);
	PyObject* start = PyObject_GetAttrString(p, "start");
	PyObject* join = PyObject_GetAttrString(p, "join");

	Py_DECREF(PyObject_CallNoArgs(start));
	Py_DECREF(PyObject_CallNoArgs(join));

	Py_DECREF(join);
	Py_DECREF(start);
	Py_DECREF(p);
	Py_DECREF(Process);
	Py_DECREF(multiprocessing);
	Py_Finalize();
}

[bstaletic]

The multiprocessing.Process, on Linux, ends up doing something like this:

pid = os.fork()
if pid == 0: os._exit()

Translated to C:

int main() {
    Py_Initialize();
    PyOS_BeforeFork();
    pid_t pid = fork();
    if (pid == 0) {
        PyOS_AfterFork_Child(); // Reinitializes stuff.
        _exit(0); // Child process exits without cleanup.
    }
    PyOS_AfterFork_Parent();
    Py_Finalize();
}

The call to _exit() happens in Lib/multiprocessing/popen_fork.py#L73

My attempts at cleaning this up resulted in even more problems.

[shihai1991]

The following locks are leaked:

1. cpython/Python/pystate.c

   Line 78 in 196d4de

   runtime->interpreters.mutex = PyThread_allocate_lock();

2. cpython/Python/pystate.c

   Line 84 in 196d4de

   runtime->xidregistry.mutex = PyThread_allocate_lock();

3. cpython/Python/pystate.c

   Line 90 in 196d4de

   runtime->main_thread = PyThread_get_thread_ident();

4. https://github.com/python/cpython/blob/master/Python/ceval.c#L810
5. https://github.com/python/cpython/blob/master/Python/import.c#L126
6. and 7. https://github.com/python/cpython/blob/master/Modules/_threadmodule.c#L597

Hi, Boris. Thanks for your report. We may receive the wrong alarm sometimes when we use valgrind.

1. cpython/Python/pystate.c

   Line 78 in 4512848

   runtime->interpreters.mutex = PyThread_allocate_lock();

   
   free in here:

   cpython/Python/pystate.c

   Line 127 in 196d4de

   FREE_LOCK(runtime->interpreters.mutex);

2. https://github.com/python/cpython/blob/main/Python/ceval.c#L741
   free in here:

   cpython/Python/pystate.c

   Line 375 in 4512848

   PyInterpreterState_Delete(PyInterpreterState *interp)

3. cpython/Python/import.c

   Line 124 in 4512848

   import_lock = PyThread_allocate_lock();

4. free in here:

   cpython/Python/import.c

   Line 156 in 4512848

   PyThread_release_lock(import_lock);

5. cpython/Modules/_threadmodule.c

   Line 613 in 4512848

   self->lock_lock = PyThread_allocate_lock();

   
   free in here:

   cpython/Modules/_threadmodule.c

   Line 74 in 4512848

   tp->tp_free((PyObject*)self);

   
   To tihs refleak, I can't get the entire traceback of newobject. But I haven't get refleak when I run ./python -m test _test_multiprocessing -R 3:3 -vv.