CommClosedError thrown by python script but not by IPython notebook

[salestial]

Hi

I get the following error if I run a dask.distributed script but no error when I run the same code in an IPython notebook:

tornado.application - ERROR - Exception in callback <functools.partial object at 0x7fea917e4ba8>
Traceback (most recent call last):
  File "/home/beowulf/miniconda2/envs/clarke/lib/python2.7/site-packages/tornado/ioloop.py", line 604, in _run_callback
    ret = callback()
  File "/home/beowulf/miniconda2/envs/clarke/lib/python2.7/site-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/beowulf/miniconda2/envs/clarke/lib/python2.7/site-packages/distributed/batched.py", line 102, in send
    raise CommClosedError

This is the code:

from distributed import Client

client = Client("127.0.0.1:8786")

def get_hostname():
    import socket
    return socket.gethostname()

def dummy(x):
    return (get_hostname(), )

jobs = client.map(dummy, range(10000))
total = client.submit(len, jobs)
total.result()

I am guessing this is happening because the script terminates before the jobs do?

Any help would be gratefully received. Thanks.

[mrocklin]

Calling total.result() will block until that future finishes. When I run your code I don't experience such an error; everything runs fine.

[salestial]

Hmm.

I am using conda with the following versions:

dask 0.14.0 py27_0
distributed 1.16.0 py27_0
ipython 5.3.0 py27_0
python 2.7.13 0
tornado 4.4.2 py27_0

Is this what you are using? Thanks.

[mrocklin]

I'm use the git-master version of dask and dask/distributed and python 3.5

…

On Wed, Mar 8, 2017 at 11:04 AM, salestial ***@***.***> wrote: Hmm. I am using conda with the following versions: dask 0.14.0 py27_0 distributed 1.16.0 py27_0 ipython 5.3.0 py27_0 python 2.7.13 0 Is this what you are using? Thanks. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#934 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszByPPeH0WyodCFcR7dYPnt4x3rIDks5rjtGngaJpZM4MW8Wb> .

[salestial]

Thanks. Will keep at it and post a solution if found.

[martindurant]

Running this now, it did work but I got a lot of these messages

distributed.scheduler - INFO - Plugin failed with exception
Traceback (most recent call last):
  File "/Users/mdurant/Downloads/distributed/distributed/scheduler.py", line 2778, in transition
    plugin.transition(key, start, finish2, *args, **kwargs)
  File "/Users/mdurant/Downloads/distributed/distributed/diagnostics/progress.py", line 286, in transition
    self.nbytes[k] -= self.scheduler.nbytes[key]
KeyError: 'dummy-c22ae48c914c9590039e92ba5daf3f8b'