Import system deadlocks in multithreading environments when performing circular imports

[Jerem2360]

Bug report

Bug description:

Hello,
By pure chance, I have found the following, which I consider as a bug for reasons that will appear in the description of said bug.

Suppose we have the following python modules:

foomodule.py:

import threading

msg = "hello from foo"


from barmodule import baz


t = threading.Thread(target=baz)
t.start()
t.join()

barmodule.py

def baz():
    import foomodule
    print('baz', foomodule.msg)

If we run this code with the command python foomodule.py, the program hangs forever. However, a normal python user would probably expect this to not happen as the thread is in principle meant to run in parallel of the code following the t.start() call.
I do know that the lock on which it is waiting is used to protect the import system from race conditions, but the point is that this deadlock is not detected and the user gets no error message or warning telling them they may have done something wrong (in this case a circular import through multiple threads), which is why I consider this as a bug.

Hope someone reviews this bug report and can fix this issue.
Regards,
Jerem2360.

CPython versions tested on:

3.12

Operating systems tested on:

Windows

[ZeroIntensity]

Hi! I'm pretty sure this is the intended behavior.

However, a normal python user would probably expect this to not happen as the thread is in principle meant to run in parallel of the code following the t.start() call.

You call join right after starting the thread, which prevents it from running in parallel. You can get around the deadlock in the reproducer by passing daemon=True and removing the call to join. Or better yet, wrap it in a __name__ check:

import threading
from barmodule import baz

msg = "hello from foo"

if __name__ == "__main__":
    t = threading.Thread(target=baz)
    t.start()
    t.join()

Circular imports can be tricky, but I don't think there's anything that's a bug here :(

[picnixz]

@hacscred Please, try to avoid being off-topic since your message does not help in this specific situation. Thank you.

[Jerem2360]

Hello,
I've dug a bit deeper into the question. That is, I modified foomodule as so:

import threading
from barmodule import baz

msg = "foo message"

print("running inside foomodule.py")

t = threading.Thread(target=baz)
t.start()
t.join()

In addition, I created a simple script in order to find out what was going on under the hoods, using python's native debugging abilities:

demo.py:

import sys
import threading


def profile(frame, event, arg):
    if event in ('c_call', 'c_return'):
        print("profile", f"in thread {threading.get_ident()}", frame, event, arg)
    return profile


def trace(frame, event, arg):
    if event in ('call', 'return'):
        print("trace", f"in thread {threading.get_ident()}", frame, event, arg)
    return trace


sys.setprofile(profile)
sys.settrace(trace)


import foomodule

running this script, I get the following output (truncated for sanity):

[...]
profile in thread 13188 <frame at 0x000001F4D31366C0, file 'D:\\Programs\\x64\\Python\\3.12\\Lib\\threading.py', line 1493, code current_thread> c_call <built-in function get_ident>
profile in thread 13188 <frame at 0x000001F4D31366C0, file 'D:\\Programs\\x64\\Python\\3.12\\Lib\\threading.py', line 1493, code current_thread> c_return <built-in function get_ident>
trace in thread 13188 <frame at 0x000001F4D31366C0, file 'D:\\Programs\\x64\\Python\\3.12\\Lib\\threading.py', line 1493, code current_thread> return <_MainThread(MainThread, started 13188)>
trace in thread 13188 <frame at 0x000001F4D34B5F20, file 'D:\\Programs\\x64\\Python\\3.12\\Lib\\threading.py', line 1155, code _wait_for_tstate_lock> call None
profile in thread 13188 <frame at 0x000001F4D34B5F20, file 'D:\\Programs\\x64\\Python\\3.12\\Lib\\threading.py', line 1169, code _wait_for_tstate_lock> c_call <built-in method acquire of _thread.lock object at 0x000001F4D34D3000>
[and then the program waits forever]

From this output, it is obvious that there is a deadlock, because the program waits on the acquire() method of a lock object.
Note that 'running inside foomodule.py' was printed exactly twice.

The main point is the following. With such a program, most programmers would expect the following:

1. foomodule is run in the main thread
2. main thread starts a new thread (lets say thread n°1) [foomodule.py, line 8, in module]
3. main thread waits for thread n°1 to finish through Thread.join call [foomodule.py, line 9, in module]
4. thread n°1 runs barmodule.baz
5. thread n°1 imports foomodule [barmodule.py, line 2, in baz]
6. thread n°1 starts a new thread (thread n°2 this time) [foomodule.py, line 8, in module]
7. thread n°1 waits for thread n°2 to finish by calling Thread.join [foomodule.py, line 9, in module]
8. thread n°2 runs barmodule.baz
9. thread n°2 imports foomodule [barmodule.py, line 2, in baz]
   etc...

I know that such code is a programming error, but the point is, what happens is not what would be expected. Instead of the previous steps resulting in an infinite number of threads being created, the actual behaviour is that the program deadlocks when thread n°1 tries to import foomodule from within barmodule.baz.

Additionnally, the if __name__ == '__main__' check doesn't always work as foomodule.py could have been imported from another python script, resulting in the code protected by this guard never being run. Also note that I am NOT trying to make my program work. I would just prefer that python fails gracefully.

To me, this feels confusing as I would have expected some kind of RecursionError but for threads to have been raised by Python. Such an error would be beneficial because it would clearly inform the programmer that they have created an infinite 'recursion'.
I suggest that, if the deadlock was to be fixed, such an error could be raised by the interpreter when a thread creates another thread with the same code as itsef which does the same, etc.. too many times, using a threshold similar to sys.getrecursionlimit on the number of times.

So it seems this is more of a logic issue than an actual bug, and maybe I should post it under another kind of issue. If so, which one ?

Hope you understand my point and can redirect me to the right place.

[picnixz]

I suggest that, if the deadlock was to be fixed, such an error could be raised by the interpreter when a thread creates another thread with the same code as itsef which does the same, etc.. too many times, using a threshold similar to sys.getrecursionlimit on the number of times.

Unfortunately, we can't be detect this easily. I don't think we can do anything and this would be the work of static analyzers to warn you about this (if they can!) or AI. The interpreter unfortunately should not protect against this because it likely can't (how can you know in advance that you will have a circular import before starting the thread? you'll need to analyze the code, which is not the role of the interpreter itself, but rather a linter).

I know that such code is a programming error, but the point is, what happens is not what would be expected.

Python cannot be foolproof everywhere. We can't prevent every programming error from misbehaving. Sorry.

Additionnally, the if __name__ == '__main__' check doesn't always work as foomodule.py could have been imported from another python script, resulting in the code protected by this guard never being run.

You're not meant to import this kind code. There is a difference between executing the project (your "main") and importing your main. If you want something like python script.py, then you likely want your script.py to be your main file and not a file whose main body would be imported and executed everytime.

---

I'm closing this one as not planned. If you need help for organizing your script, you can ask on https://discuss.python.org/c/help/7.