Update pyav_reader.py to make generator threadsafe #395
Conversation
I had an issue where code using pyav_reader.py would randomly break and throw a "ValueError: Generator Already Executing Error"; the stack indicated that the function _gen_frames was the culprit; I added a function and a decorator (line for line copied from the sources below with one change, re-naming "next" to "__next__") that makes a non-threadsafe generator threadsafe. This resolved the issue on both Windows and Mac. Sources: https://anandology.com/blog/using-iterators-and-generators/ https://stackoverflow.com/questions/41194726/python-generator-thread-safety-using-keras
| @@ -2,6 +2,7 @@ | |||
| unicode_literals) | |||
|
|
|||
| import numpy as np | |||
| import threading | |||
There was a problem hiding this comment.
Forgot to add the import for threading to the original pull request!
Sometimes the self.it.next() method was breaking; looks like that method might be an artifact of older versions of Python? Fixed it to modern spec to prevent future errors.
|
"powercycled" to get CI. |
Codecov Report
@@ Coverage Diff @@
## main #395 +/- ##
=======================================
Coverage ? 63.01%
=======================================
Files ? 31
Lines ? 4597
Branches ? 0
=======================================
Hits ? 2897
Misses ? 1700
Partials ? 0 Continue to review full report at Codecov.
|
| def threadsafe_generator(f): | ||
| """A decorator that takes a generator function and makes it thread-safe. | ||
| """ | ||
| def g(*a, **kw): |
There was a problem hiding this comment.
Can you add functools.wraps(f) here as well?
|
@BakudanKame Sorry for the very slow review! This seems reasonable, but I am a bit wary of tacking locks onto things, is there a reasonable way to let the user opt-into the locking/thread safety? |
I had an issue where code using pyav_reader.py would randomly break and throw a "ValueError: Generator Already Executing Error"; the stack indicated that the function _gen_frames was the culprit; I added a function and a decorator (line for line copied from the sources below with one change, re-naming "next" to "next") that makes a non-threadsafe generator threadsafe. This resolved the issue on both Windows and Mac. I thought I'd send it over to you guys for your perusal! Thanks for the great package!
The original error message:
Sources:
https://anandology.com/blog/using-iterators-and-generators/
https://stackoverflow.com/questions/41194726/python-generator-thread-safety-using-keras