Race condition and crash when embedding multi-thread script

[OleksiyMarkovets]

BPO	24544
Nosy	@iritkatriel
Files	main.cpp: c++ part main.py: python part

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-09-09.18:31:46.524>
created_at = <Date 2015-07-01.13:05:29.741>
labels = ['extension-modules', 'type-crash']
title = 'Race condition and crash when embedding multi-thread script'
updated_at = <Date 2021-09-09.18:31:46.523>
user = 'https://bugs.python.org/OleksiyMarkovets'

bugs.python.org fields:

activity = <Date 2021-09-09.18:31:46.523>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-09.18:31:46.524>
closer = 'iritkatriel'
components = ['Extension Modules']
creation = <Date 2015-07-01.13:05:29.741>
creator = 'Oleksiy Markovets'
dependencies = []
files = ['39840', '39841']
hgrepos = []
issue_num = 24544
keywords = []
message_count = 3.0
messages = ['246035', '246036', '401508']
nosy_count = 2.0
nosy_names = ['Oleksiy Markovets', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue24544'
versions = ['Python 2.7']

[OleksiyMarkovets]

INTRODUCTION

While embedding python script in c++ application I faced random crashes
with error:

Fatal Python error: Py_EndInterpreter: not the last thread
Aborted

Even though I explicitly join each thread created with *threading*
module. Please see attachment, this is simplest python script +
c++ code which demonstrates problem.

By using threading.setprofiler I was able to make problem appear each run.
If you uncomment time.sleep(1) problem won't reproduce.

Note:
In main.cpp I manually create separate interpreter because in real-life
application is multi-threaded and different thread uses different interpreters
for at least some sand-boxing.

INVESTIGATION

I did some investigation, and here is what I found out:

• Each new thread is started by calling *thread_PyThread_start_new_thread* from threadmodule.c.
  Basically this function creates new thread which executes *t_bootstrap* and adds this
  thread to interpreter's thread list.

• *t_bootstrap* executes python function *Thread.__bootstrap* and when it's done, removes
  thread from interpreter's thread list.

• *Thread.__bootstrap* runs *Thread.run()* (actually python code which should be executed
  in separate thread) and when it's done calls *Thread.__stop* (which by mean of condition
  variable sets Boolean flag Thread.__stopped)

• *Thread.join* doesn't wait thread to exit, it only waits for *Thread.__stopped* flag
  to be set.

So here is race condition:
When main thread finished *Thread.join* it's only guaranteed that *Thread.run*
is finished, but *t_bootstrap* still may be running (interpreter's thread list
is not cleared).

POSSIBLE SOLUTION

To call Thread.__stop from *t_bootstrap* (instead of Thread.__bootstrap) after
removing thread from interpreter's thread list.

Or do not use detached threads and call something like *pthread_join* in *Thread.join*.
Which is IMHO much clearer approach but also requires much more efforts.

[OleksiyMarkovets]

attached file

[iritkatriel]

Python 2.7 is no longer maintained. Please create a new issue if you are having problems with this in a current version (>= 3.9).