Should worker_client execute rejoin as well

[ameetshah1983]

We have a task which calls a function executing worker_client. As this function gets called few times as part of the task execution, looks like we get an error:

File "e:\anaconda\python\lib\site-packages\distributed\worker_client.py", line 46, in worker_client
secede() # have this thread secede from the thread pool
File "e:\anaconda\python\lib\site-packages\distributed\threadpoolexecutor.py", line 104, in secede
thread_state.executor._threads.remove(threading.current_thread())
KeyError: <Thread(ThreadPool worker 0, started daemon 8556)>

Added rejoin after the "with" block and then it works fine. Should the work_client rejoin or have option to rejoin for such cases. Please let me know if this should be handled in any other way.

Note - Have left separate_thread option as True as not sure if setting it to False can result in the deadlock descibed in the documentation which is why we are utilizing worker_client. We are running single worker for each core with single thread.

[mrocklin]

Yes, I think that it probably makes sense for the worker_client context manager to rejoin. My apologies for my comment in the other issue where I erroneously assumed that this was already the case.

…

On Thu, May 31, 2018 at 10:25 PM, ameetshah1983 ***@***.***> wrote: We have a task which calls a function executing worker_client. As this function gets called few times as part of the task execution, looks like we get an error: File "e:\anaconda\python\lib\site-packages\distributed\worker_client.py", line 46, in worker_client secede() # have this thread secede from the thread pool File "e:\anaconda\python\lib\site-packages\distributed\threadpoolexecutor.py", line 104, in secede thread_state.executor._threads.remove(threading.current_thread()) KeyError: <Thread(ThreadPool worker 0, started daemon 8556)> Added rejoin after the "with" block and then it works fine. Should the work_client rejoin or have option to rejoin for such cases. Please let me know if this should be handled in any other way. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2021>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszJSvnO0QHUTC-PH9CjpQmpyb1_Auks5t4KYGgaJpZM4UWACo> .

[mrocklin]

Fixed in #2041