Don't block event loop in Worker.close waiting for executor to shut down

[gjoseph92]

distributed/distributed/worker.py

Lines 1517 to 1521 in 1cbee7f

	if isinstance(executor, ThreadPoolExecutor):
	executor._work_queue.queue.clear()
	executor.shutdown(wait=executor_wait, timeout=timeout)
	else:
	executor.shutdown(wait=executor_wait)

shutdown is a blocking call that generally calls join on the thread/process. If the executor takes a long time to shut down (say there's a function running that's itself blocked on something), this can block the worker event loop for 30s, or whatever the timeout is.

This is particularly important for our test suite, because the worker event loop is actually the only event loop.

I discovered this trying to run a test something like

event = distributed.Event()
f = client.submit(event.wait, workers=[a.address])
t = asyncio.create_task(a.close())
await asyncio.sleep(1)
await event.set()  # this hangs!
# because the whole event loop is blocked waiting for the ThreadPoolExecutor to shut down,
# which is waiting for the event to be set... which is waiting for the event loop to be free

This probably won't affect real-life clusters that much, but would be good to clean up.

cc @graingert

[mrocklin]

See also #6091 maybe

…

On Thu, Apr 28, 2022 at 3:03 PM Gabe Joseph ***@***.***> wrote: https://github.com/dask/distributed/blob/1cbee7f18717f060c65b2ac44445745135c8dd46/distributed/worker.py#L1517-L1521 shutdown is a blocking call that generally calls join on the thread/process. If the executor takes a long time to shut down (say there's a function running that's itself blocked on something), this can block the worker event loop for 30s, or whatever the timeout is. This is particularly important for our test suite, because the worker event loop is actually the *only* event loop. I discovered this trying to run a test something like event = distributed.Event()f = client.submit(event.wait, workers=[a.address])t = asyncio.create_task(a.close())await asyncio.sleep(1)await event.set() # this hangs!# because the whole event loop is blocked waiting for the ThreadPoolExecutor to shut down,# which is waiting for the event to be set... which is waiting for the event loop to be free This *probably* won't affect real-life clusters that much, but would be good to clean up. cc @graingert <https://github.com/graingert> — Reply to this email directly, view it on GitHub <#6239>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTAGMPXPA65JR2XKCLTVHLVIBANCNFSM5UTTGKJA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[gjoseph92]

@mrocklin thanks, #6091 would fix this.