-
-
Notifications
You must be signed in to change notification settings - Fork 281
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TypeError: cannot pickle '_thread.RLock' object with ProcessPoolExecutor #429
Comments
we're also hitting this but using Here is @davetapley's adapted MRE: from multiprocessing import get_context
from dataclasses import dataclass
from typing import Callable, Generic, TypeVar
from tenacity import (retry, retry_if_exception_type, stop_after_attempt,
wait_fixed)
I = TypeVar('I')
O = TypeVar('O')
class IOException(Exception):
...
@dataclass
class Job(Generic[I, O]):
f: Callable[[I], O]
@retry(
retry=retry_if_exception_type(IOException),
wait=wait_fixed(0.1),
stop=stop_after_attempt(10),
#reraise=True,
)
def __call__(self, x: I):
return self.f(x)
def good_job(x: int) -> int:
return x * 2
def fail_job(x: int) -> int:
raise IOException('nope')
data = [1, 2, 3, 4, 5]
if __name__ == '__main__':
with get_context('spawn').Pool() as pool:
for result in pool.imap_unordered(Job(good_job), data):
print(result)
with get_context('spawn').Pool() as pool:
for result in pool.imap_unordered(Job(fail_job), data):
print(result) output with
output with
|
^ the issue also seems unrelated to the use of |
The retry error from tenacity uses threading locks under the hood, which cannot be pickled. Specifying reraise=True avoids this particular codepath. See upstream issues: * jd/tenacity#429 * jd/tenacity#182 * jd/tenacity#147
I also encountered this problem. @spolloni's solution |
Here's a weird one.
I'm using
ProcessPoolExecutor
and sometimes that causes in contention on my databases.I already use
tenacity
elsewhere, so I figure can just wrap inretry_if_exception_type(IOException)
.But when
IOException
is thrown I getTypeError: cannot pickle '_thread.RLock' object
.MRE:
Output:
The text was updated successfully, but these errors were encountered: