restart time out and upload_file crash

[apatlpo]

I updated dask/xarray via conda yesterday (to distributed-1.21.4) and started having issues with workers not starting on an hpc system.
Research eventually brought me to 31c470e which I tried.
It seems to solve some issues i.e.:

• workers are starting
• client restarts still appear to have some issues:

client = Client(scheduler_file=os.path.expanduser('/home1/scratch/aponte/dask/scheduler.json'))
client.restart()

leads to:

distributed.client - ERROR - Restart timed out after 3.000000 seconds

I can work however but have some issues when trying to upload files (I realize this may be another issue):

client.upload_file('/home1/datahome/aponte/iwave_sst/hw/utils.py')

leads to:

distributed.utils - ERROR - [Errno 2] No such file or directory: '/home1/scratch/aponte/dask/worker-oft8kj37/utils.py'
Traceback (most recent call last):
  File "/home1/datahome/aponte/.miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/utils.py", line 238, in f
    result[0] = yield make_coro()
  File "/home1/datahome/aponte/.miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home1/datahome/aponte/.miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)

The /home1/scratch/aponte/dask/worker-oft8kj37/ does not exist indeed.

When I look for oft8kj37 in log files I get the following.
The last hit may be relevant:

iwave_sst/datarmor% grep oft8kj37 dask-*
dask-scheduler.o958818:distributed.core - ERROR - [Errno 2] No such file or directory: '/home1/scratch/aponte/dask/worker-oft8kj37/utils.py'
dask-scheduler.o958818:FileNotFoundError: [Errno 2] No such file or directory: '/home1/scratch/aponte/dask/worker-oft8kj37/utils.py'
dask-worker.o958819:distributed.worker - INFO -       Local Directory: /home1/scratch/aponte/dask/worker-oft8kj37
dask-worker.o958819:distributed.core - ERROR - [Errno 2] No such file or directory: '/home1/scratch/aponte/dask/worker-oft8kj37/utils.py'
dask-worker.o958819:FileNotFoundError: [Errno 2] No such file or directory: '/home1/scratch/aponte/dask/worker-oft8kj37/utils.py'
dask-worker.o958819:distributed.core - ERROR - [Errno 2] No such file or directory: '/home1/scratch/aponte/dask/worker-oft8kj37/utils.py'
dask-worker.o958819:FileNotFoundError: [Errno 2] No such file or directory: '/home1/scratch/aponte/dask/worker-oft8kj37/utils.py'
dask-worker.o958822:distributed.diskutils - WARNING - Found stale lock file and directory '/home1/scratch/aponte/dask/worker-oft8kj37', purging

[mrocklin]

Can you try starting your client with the following:

client = Client(..., timeout='20s')

and then seeing how the restart does?

[mrocklin]

For the file issue I'm not surprised to see things not work well on an NFS system. See http://dask.pydata.org/en/latest/setup/hpc.html#no-local-storage

If you have an NFS then maybe just use normal techniques like PYTHONPATH or installing modules rather than use upload_file

[apatlpo]

With 20s:

distributed.client - ERROR - Restart timed out after 20.000000 seconds

With 60s:

distributed.batched - INFO - Batched Comm Closed: in <closed TCP>: BrokenPipeError: [Errno 32] Broken pipe
tornado.application - ERROR - Exception in callback <bound method Client._heartbeat of <Client: scheduler='tcp://10.148.0.106:8786' processes=33 cores=132>>
Traceback (most recent call last):
  File "/home1/datahome/aponte/.miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/ioloop.py", line 1209, in _run
    return self.callback()
  File "/home1/datahome/aponte/.miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py", line 846, in _heartbeat
    self.scheduler_comm.send({'op': 'heartbeat'})
  File "/home1/datahome/aponte/.miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/batched.py", line 106, in send
    raise CommClosedError

the message repeats itself several times

[mrocklin]

How are you starting your workers? What kind of system are you on?

[apatlpo]

I am on a cluster PBS : https://wwz.ifremer.fr/pcdm/Equipement
I run ./launch-dask.sh ${N_WORK_NODES} following https://pangeo-data.github.io/pangeo/setup_guides/cheyenne.html
It was working fine until a recent update xarray/dask/...

[mrocklin]

It was working fine until a recent update xarray/dask/...

Would you be willing to use git bisect to find the cause? "No" is a fine answer

[apatlpo]

I started doing this but am right away confused.

The working version is dask 0.15.4 and distributed 1.19.3

The failing version is dask 0.17.2 and distributed 1.18.1+288.g31c470e

Isn't 31c470e based on the latest distributed version?

[mrocklin]

Correct. You would only be bisecting in the distributed repository. You would no longer use 31c470e at all, instead you would bound your search by a commit that you knew worked (like version 1.19.3) and a commit that you knew did not work (like version 1.21.4) and then you would use git bisect to search within that range to find the offending commit. This process is very useful, but can be confusing at first.

…

On Wed, Mar 28, 2018 at 8:53 AM, Aurélien Ponte ***@***.***> wrote: I started doing this but am right away confused. The working version is dask 0.15.4 and distributed 1.19.3 The failing version is dask 0.17.2 and distributed 1.18.1+288.g31c470e Isn't 31c470e <31c470e> based on the latest distributed version? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1869 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszDm3Lb4BTjv7G2oi3GHRkfuGmeaLks5ti4fGgaJpZM4S-iLv> .

[apatlpo]

Am I supposed to this simultaneously for dask AND distributed?
If yes, I am not sure how I can keep consistent versions of both libraries while bisecting.

[mrocklin]

Hopefully there is enough flexibility that everything will work smoothly with a single recent version of Dask. Also, you have no obligation to do this. It would be useful, but if it proves to be too difficult then we can probably find the issue some other way with more time.

…

On Wed, Mar 28, 2018 at 9:29 AM, Aurélien Ponte ***@***.***> wrote: Am I supposed to this simultaneously for dask AND distributed? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1869 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszBsCZDf9gIZMharnnDTYxHL8gdl1ks5ti5AwgaJpZM4S-iLv> .

[apatlpo]

Hum, distributed version 1.19.3 does not seem to be compatible with latest dask.

The combination does not even produce a scheduler.json

The following error on the scheduler side occurs:

  File "/home1/datahome/aponte/.miniconda3/envs/pangeo/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home1/datahome/aponte/.miniconda3/envs/pangeo/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home1/datahome/aponte/.miniconda3/envs/pangeo/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home1/datahome/aponte/distributed/distributed/cli/dask_mpi.py", line 57, in main
    services=services)
  File "/home1/datahome/aponte/distributed/distributed/scheduler.py", line 376, in __init__
    **kwargs)
  File "/home1/datahome/aponte/distributed/distributed/node.py", line 37, in __init__
    deserialize=deserialize, io_loop=self.io_loop)
  File "/home1/datahome/aponte/distributed/distributed/core.py", line 117, in __init__
    pc = PeriodicCallback(self.monitor.update, 500, io_loop=self.io_loop)
TypeError: __init__() got an unexpected keyword argument 'io_loop'
Terminated

[apatlpo]

argh, this is a distributed-tornado compatibility issue in fact, see #1172

The bisect seems difficult to me

[apatlpo]

Unless you see an easier alternative, I'll try a manual bisect with conda install to narrow down where the problem arise