Client.close raises StreamClosedError

[jakirkham]

Appears the running Client.close consistently runs into some sort of hang up (shown below). This happens in Distributed 1.21.8, but not in Distributed 1.21.6. Context for this provided in the linked Gist, which contains the Jupyter Notebook it showed up in and the environment.

Exception ignored in: <generator object add_client at 0x11e4d1f10>
RuntimeError: generator ignored GeneratorExit
Future exception was never retrieved
future: <Future finished exception=CommClosedError('in <closed TCP>: Stream is closed',)>
Traceback (most recent call last):
  File "/zopt/conda2/envs/test/lib/python3.6/site-packages/distributed/comm/tcp.py", line 179, in read
    n_frames = yield stream.read_bytes(8)
  File "/zopt/conda2/envs/test/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/zopt/conda2/envs/test/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/zopt/conda2/envs/test/lib/python3.6/site-packages/distributed/comm/tcp.py", line 200, in read
    convert_stream_closed_error(self, e)
  File "/zopt/conda2/envs/test/lib/python3.6/site-packages/distributed/comm/tcp.py", line 128, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed

ref: https://gist.github.com/jakirkham/b9c75e2f7fabdee5f19db8cc3bff0d9e
ref: #1963

[macks22]

I'm getting the same issue, except I also get this issue with distributed==1.21.6. The issue goes away when I downgrade tornado to 4.5.1 (which I chose since it was the minimum supported version per the dask requirements file).

Versions:

dask-0.17.4
distributed-1.21.8
tornado-5.0.2

Minimum example:

import time
from dask.distributed import Client, as_completed


def square(i):
    return i ** 2

with Client(n_workers=2) as client:
    futures = [client.submit(square, i) for i in range(10)]
    for f, result in as_completed(futures, with_results=True):
        print(result)
        time.sleep(0.1)  # just to make it more obvious what's happening

    time.sleep(1)

which first outputs:

64
0
36
16
4
81
49
25
9
1

as expected, but then also:

Exception ignored in: <generator object add_client at 0x120368258>
RuntimeError: generator ignored GeneratorExit
[ERROR][2018-05-08 15:33:24,001]: Future exception was never retrieved
future: <Future finished exception=CommClosedError('in <closed TCP>: Stream is closed',)>
Traceback (most recent call last):
  File "/Users/vru959/anaconda2/envs/py3k/lib/python3.6/site-packages/distributed/comm/tcp.py", line 179, in read
    n_frames = yield stream.read_bytes(8)
  File "/Users/vru959/anaconda2/envs/py3k/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/vru959/anaconda2/envs/py3k/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/vru959/anaconda2/envs/py3k/lib/python3.6/site-packages/distributed/comm/tcp.py", line 200, in read
    convert_stream_closed_error(self, e)
  File "/Users/vru959/anaconda2/envs/py3k/lib/python3.6/site-packages/distributed/comm/tcp.py", line 128, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed

[jakirkham]

So there are other cases where StreamClosedError does get raised. They do need to be fixed. This issue is just showing that now simply calling close will trigger a StreamClosedError. I think you have found a different case where this occurs, @macks22. So it would be reasonable to move your use case to it's own issue. Does that sound reasonable?

[macks22]

@jakirkham I can see your concern -- I definitely don't want to convolute two issues. However, I think this is the same problem. My example simply uses the context manager for the Client, whose __exit__ just calls Client.close, which is what triggers the exception. This leads me to believe this is the same issue, and hence, that my comment should remain here and not be filed as another issue.

[jakirkham]

Ok, thanks for clarifying. :)

[macks22]

@jakirkham did you ever find a resolution for this? I haven't upgraded distributed versions in a while and wondering if you were aware of a fix being introduced in later versions. Thanks!

[jakirkham]

Thanks for the reminder.

Have not had time to look into the cause (or fix it) personally. Though can confirm this issue is still present with Distributed 1.22.1 (latest).

[cicdw]

I dug into this a little bit and it looks like this might be caused by this line which is triggered by stopping the LoopRunner attached to the LocalCluster.

[mrocklin]

Yeah, so what happens here is that the IOLoop stops, this severs a bunch of connections, and they raise errors. In many cases these are caught appropriately, but in some they aren't.

Pragmatically this may be resolved for this particular issue here: #2237

The connection that was complaining was the Client's connection to the scheduler. We now wait a little bit of time to see if the scheduler's "all finished now" message arrives in time, which results in a quieter experience, at least in the example provided here.