-
-
Notifications
You must be signed in to change notification settings - Fork 31.6k
test_threaded_import: KeyError ignored in _get_module_lock.<locals>.cb #75253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
See bpo-bpo-30891 and bpo-30876 for recent changes in importlib. 1:33:30 [312/406/2] test_threaded_import passed (30 sec) -- running: test_decimal (826 sec), test_set (84 sec), test_mmap (1151 sec)
beginning 6 repetitions
123456
...Exception ignored in: <function _get_module_lock.<locals>.cb at 0x000000813CA18EB8>
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 176, in cb
KeyError: ('random',)
... |
Oh, it's easy to reproduce the bug on the master branch. Example on Linux: haypo@selma$ ./python -m test test_threaded_import
Run tests sequentially
0:00:00 load avg: 0.16 [1/1] test_threaded_import
Exception ignored in: <function _get_module_lock.<locals>.cb at 0x7f53201e2f70>
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 176, in cb
KeyError: ('random',)
1 test OK. Total duration: 2 sec |
The problem is the weakref callback of _module_locks. It seems like the callback is called after a second thread replaced _module_locks[name]
|
I wrote #3029 which seems to fix the issue. I don't know importlib well enough to understand why we need a weak reference to a lock. My PR adds a second lock per module lock (!) to be able to wait until _module_locks[name] is deleted when a thread detects that _module_locks[name] exists and the associated lock was destroyed. |
I think the solution can be simpler. PR 3033 uses the global import lock for guarding modification of the _module_locks dictionary. |
Serhiy: "I think the solution can be simpler. PR 3033 uses the global import lock for guarding modification of the _module_locks dictionary." I tried exactly that, but it wasn't enough. But your PR is correct because it also checks the current value of _module_locks[name] using the callback parameter. I disliked my own PR, so I'm happy that someone else wrote a simpler fix! (I already abandonned my PR.) |
Thank you for review Victor and Nick. |
I don't think the changes created the bug, but I do think the new tests to provoke the other race conditions made it easier for the test suite to hit this race as well. |
Oh ok, that's a good news! |
A recent numpy threaded import bug report that appears in 3.6.1 but is absent in 3.6.3: numpy/numpy#9931 So that's additional evidence that this really was a pre-existing race condition that the new tests for the SystemError cases revealed. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: