PyThreadState_Next not thread safe

[jpe]

BPO	1021318
Nosy	@pitrou, @vstinner, @devdanzin, @jd, @ericsnowcurrently, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2004-09-02.16:56:47.000>
labels = ['interpreter-core', '3.9']
title = 'PyThreadState_Next not thread safe'
updated_at = <Date 2020-05-15.01:09:40.629>
user = 'https://bugs.python.org/jpe'

bugs.python.org fields:

activity = <Date 2020-05-15.01:09:40.629>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2004-09-02.16:56:47.000>
creator = 'jpe'
dependencies = []
files = []
hgrepos = []
issue_num = 1021318
keywords = []
message_count = 14.0
messages = ['60562', '82072', '114376', '358187', '358188', '358189', '358190', '358191', '358192', '358261', '358262', '358347', '364094', '368895']
nosy_count = 7.0
nosy_names = ['jpe', 'pitrou', 'vstinner', 'ajaksu2', 'jd', 'eric.snow', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue1021318'
versions = ['Python 3.9']

[jpe]

I can't convince myself that PyThreadState_Next is
thread safe even if the GIL is held because
PyGILState_Release can delete a thread state after
releasing the GIL if the gilstate_counter reaches 0.
So another thread could be deleting a thread state my
thread is referencing.

If the GIL was held when the thread state was deleted,
the I think the next function would be thread safe
because new thread states are always added to the front
of the list.

If these functions should work when the GIL is not
held, the head_mutex in pystate.c probably needs to be
exposed in some manner so code can lock it, iterate
through the threads, and then unlock it.

[devdanzin]

Can someone take a look a this and either close or propose what to do?

[BreamoreBoy]

No reply to msg82072.

[jd]

This is actually true and it's quite easy to produce a bug that triggers a segmentation fault when iterating over a large list of threads create dynamically.

That makes PyThreadState_Next barely usable for any program that has a few threads. The head mutex needs to be locked to be able to iterate safely over the list of thread.