Losing all workers on cluster

[BrendanMartin]

I'm not sure how much detail to put here, or if this is the right place, but I am attempting to run dask_searchcv on a Kubernetes cluster and whenever I run the grid search all of my workers get killed immediately.

From the logs on the cluster I have:

distributed.scheduler - INFO - Register tcp://10.16.1.9:41940

distributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.1.9:41940

distributed.scheduler - INFO - Register tcp://10.16.0.9:46410

distributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.0.9:46410

distributed.scheduler - INFO - Register tcp://10.16.2.9:38646

distributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.2.9:38646

distributed.scheduler - INFO - Remove worker tcp://10.16.2.9:38646

distributed.scheduler - INFO - Remove worker tcp://10.16.1.9:41940

distributed.scheduler - INFO - Remove worker tcp://10.16.0.9:46410

distributed.scheduler - INFO - Lost all workers

I have a cluster of 3 CPUs and 12G RAM, and so 3 workers are spawned by default.

One thing I noticed, not sure if it's related, but when I run the grid search and look at the workers, there's always one worker that's at 90%+ utilization while the others are only around 5%.

[mrocklin]

No thoughts on what is going on. Some things to check:

1. run client.get_versions(check=True)
2. look at worker logs to see if they give information about why they are dying.

[mrocklin]

Any further information here @BrendanMartin ?

[BrendanMartin]

I ran
client.get_versions(check=True)
And got this:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-29-4a95fbecd8f8> in <module>()
----> 1 client.get_versions(check=True)

~\Anaconda3\lib\site-packages\distributed\client.py in get_versions(self, check)
   2957                 raise ValueError("Mismatched versions found\n"
   2958                                  "\n"
-> 2959                                  "%s" % ('\n\n'.join(errs)))
   2960 
   2961         return result

ValueError: Mismatched versions found

blosc
+------------------------+---------+
|                        | version |
+------------------------+---------+
| client                 | None    |
| scheduler              | 1.5.1   |
| tcp://10.16.1.13:43150 | 1.5.1   |
| tcp://10.16.1.14:36169 | 1.5.1   |
+------------------------+---------+

bokeh
+------------------------+---------+
|                        | version |
+------------------------+---------+
| client                 | 0.12.4  |
| scheduler              | 0.12.14 |
| tcp://10.16.1.13:43150 | 0.12.14 |
| tcp://10.16.1.14:36169 | 0.12.14 |
+------------------------+---------+

cloudpickle
+------------------------+---------+
|                        | version |
+------------------------+---------+
| client                 | 0.2.2   |
| scheduler              | 0.5.2   |
| tcp://10.16.1.13:43150 | 0.5.2   |
| tcp://10.16.1.14:36169 | 0.5.2   |
+------------------------+---------+

dask
+------------------------+---------+
|                        | version |
+------------------------+---------+
| client                 | 0.17.0  |
| scheduler              | 0.17.1  |
| tcp://10.16.1.13:43150 | 0.17.2  |
| tcp://10.16.1.14:36169 | 0.17.2  |
+------------------------+---------+

distributed
+------------------------+---------+
|                        | version |
+------------------------+---------+
| client                 | 1.21.3  |
| scheduler              | 1.21.1  |
| tcp://10.16.1.13:43150 | 1.21.1  |
| tcp://10.16.1.14:36169 | 1.21.1  |
+------------------------+---------+

msgpack
+------------------------+---------+
|                        | version |
+------------------------+---------+
| client                 | 0.4.8   |
| scheduler              | 0.5.5   |
| tcp://10.16.1.13:43150 | 0.5.5   |
| tcp://10.16.1.14:36169 | 0.5.5   |
+------------------------+---------+

numpy
+-----------+---------+
|           | version |
+-----------+---------+
| client    | 1.14.2  |
| scheduler | 1.14.1  |
+-----------+---------+

toolz
+------------------------+---------+
|                        | version |
+------------------------+---------+
| client                 | 0.8.2   |
| scheduler              | 0.9.0   |
| tcp://10.16.1.13:43150 | 0.9.0   |
| tcp://10.16.1.14:36169 | 0.9.0   |
+------------------------+---------+

I'm not sure what this means.

The worker logs don't say much. Here's the full log for all workers:

distributed.scheduler - INFO - Register tcp://10.16.1.10:42285

distributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.1.10:42285

distributed.scheduler - INFO - Worker 'tcp://10.16.1.11:36934' failed from closed comm: in <closed TCP>: Stream is closed

distributed.scheduler - INFO - Remove worker tcp://10.16.1.11:36934

distributed.scheduler - INFO - Worker 'tcp://10.16.1.10:42285' failed from closed comm: in <closed TCP>: Stream is closed

distributed.scheduler - INFO - Remove worker tcp://10.16.1.10:42285

distributed.scheduler - INFO - Worker 'tcp://10.16.1.9:35900' failed from closed comm: in <closed TCP>: Stream is closed

distributed.scheduler - INFO - Remove worker tcp://10.16.1.9:35900

distributed.scheduler - INFO - Lost all workers

distributed.scheduler - INFO - Register tcp://10.16.2.7:44208

distributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.2.7:44208

distributed.scheduler - INFO - Register tcp://10.16.0.8:35685

distributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.0.8:35685

distributed.scheduler - INFO - Register tcp://10.16.1.12:33932

distributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.1.12:33932

distributed.scheduler - INFO - Receive client connection: Client-951d804c-4580-11e8-b580-5cf3707e6b64

distributed.scheduler - ERROR - error from worker tcp://10.16.0.8:35685: 'NoneType' object has no attribute 'get'

distributed.scheduler - ERROR - error from worker tcp://10.16.0.8:35685: 'NoneType' object has no attribute 'get'

distributed.scheduler - INFO - Worker 'tcp://10.16.1.12:33932' failed from closed comm: in <closed TCP>: Stream is closed

distributed.scheduler - INFO - Remove worker tcp://10.16.1.12:33932

distributed.scheduler - INFO - Worker 'tcp://10.16.2.7:44208' failed from closed comm: in <closed TCP>: Stream is closed

distributed.scheduler - INFO - Remove worker tcp://10.16.2.7:44208

distributed.scheduler - INFO - Worker 'tcp://10.16.0.8:35685' failed from closed comm: in <closed TCP>: Stream is closed

distributed.scheduler - INFO - Remove worker tcp://10.16.0.8:35685

distributed.scheduler - INFO - Lost all workers

distributed.scheduler - INFO - Register tcp://10.16.0.9:35376

distributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.0.9:35376

distributed.scheduler - INFO - Register tcp://10.16.1.13:38442

distributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.1.13:38442

distributed.scheduler - INFO - Register tcp://10.16.1.14:35026

distributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.1.14:35026

distributed.scheduler - INFO - Receive client connection: Client-3078348c-4583-11e8-b580-5cf3707e6b64

distributed.scheduler - INFO - Remove worker tcp://10.16.1.14:35026

distributed.scheduler - INFO - Remove worker tcp://10.16.0.9:35376

distributed.scheduler - INFO - Remove worker tcp://10.16.1.13:38442

distributed.scheduler - INFO - Lost all workers

distributed.scheduler - INFO - Register tcp://10.16.0.9:33457

distributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.0.9:33457

distributed.scheduler - INFO - Remove worker tcp://10.16.0.9:33457

distributed.scheduler - INFO - Lost all workers

I'm getting this in my notebook when this happens:
KilledWorker: ('tfidfvectorizer-c00cd7b22afb39cd243d3e56ee9dae7d', 'tcp://10.16.0.9:41769')

Is there some place I can get more information?

[martindurant]

The output shows clearly that you have mismatched versions in, well, almost everything. It is not surprising that the dask processes are having trouble talking to one-another. Since this is a kuberneters cluster, one would expect the versions all to match if they are drawn from the same image. You should look into how you are invoking your cluster, and also make sure that the client (which I don't know if it on the cluster or not) also matches versions.

[BrendanMartin]

Yeah, I'm not really sure why it's happening. I'm essentially doing the following:
Making cluster on Google
gcloud container clusters create cluster1

Installing helm

kubectl --namespace kube-system create sa tiller
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller

Installing dask

helm install stable/dask

Installing packages on cluster

helm upgrade nobby-robin stable/dask -f c:\users\brendan\helm\config.yml

where config.yml is

worker:
  env:
    - name: EXTRA_CONDA_PACKAGES
      value: gcsfs>=0.1.0 dask_searchcv -c conda-forge
    - name: EXTRA_PIP_PACKAGES
      value: gcsfs>=0.1.0 dask_searchcv --upgrade
jupyter:
  enabled: false

And then connecting to the Client with dask.

Is there something I could be missing?

[mrocklin]

The version of dask on the machine where you create the client is quite old. You should update it using pip install dask[complete] or conda install dask. The version of dask on the machine that you use to create the client needs to be the same version that you use on your cluster.

…

On Sat, Apr 21, 2018 at 1:15 PM, Brendan Martin ***@***.***> wrote: Yeah, I'm not really sure why it's happening. I'm essentially doing the following: Making cluster on Google gcloud container clusters create cluster1 Installing helm kubectl --namespace kube-system create sa tiller kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller helm init --service-account tiller Installing dask helm install stable/dask Installing packages on cluster helm upgrade nobby-robin stable/dask -f c:\users\brendan\helm\config.yml where config.yml is env: - name: EXTRA_CONDA_PACKAGES value: gcsfs>=0.1.0 dask_searchcv -c conda-forge - name: EXTRA_PIP_PACKAGES value: gcsfs>=0.1.0 dask_searchcv --upgrade jupyter: enabled: false And then connecting to the Client with dask. Is there something I could be missing? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1914 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszEof4v-gZSiYogx1akzIEf1ELnQJks5tq2k7gaJpZM4TQB5Z> .

[BrendanMartin]

Is there a straightforward way to upgrade the cluster packages?

[mrocklin]

Different clusters have different ways of doing this. If you are using a kubernetes cluster then presumably you are using docker and so might use a different docker image. However given the message from above it may be that the version of dask on your cluster is quite up to date while the version where you are running your client is old.

…

On Sun, Apr 22, 2018 at 7:44 PM, Brendan Martin ***@***.***> wrote: Is there a straightforward way to upgrade the cluster packages? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1914 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszIR8NiLQtzdrM5t95TTCtn_UALMEks5trRXqgaJpZM4TQB5Z> .

[BrendanMartin]

dask seems to be installed on the cluster via the stable/dask helm chart.

I updated all of my client libraries, but now they are more recent versions than stable/dask, so I have mismatched versions in the opposite way now. That's why I was wondering how I might update the cluster.

[mrocklin]

The config file for the helm chart includes values for docker images. These images are built from the Dockerfiles in https://github.com/dask/dask-docker

…

On Mon, Apr 23, 2018 at 5:46 PM, Brendan Martin ***@***.***> wrote: dask seems to be installed on the cluster via the stable/dask helm chart. I updated all of my client libraries, but now they are more recent versions than stable/dask, so I have mismatched versions in the opposite way now. That's why I was wondering how I might update the cluster. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1914 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszPZ84RsLvBm1bFFEzOhDBF5vSSwHks5trkuwgaJpZM4TQB5Z> .

[BrendanMartin]

Do those always reflect the latest version? Because when I use helm install stable/dask, and after updating all of my client packages, this is the check versions now:

ValueError: Mismatched versions found

bokeh
+-----------------------+---------+
|                       | version |
+-----------------------+---------+
| client                | 0.12.13 |
| scheduler             | 0.12.14 |
| tcp://10.16.0.7:45136 | 0.12.14 |
| tcp://10.16.1.8:36913 | 0.12.14 |
| tcp://10.16.2.5:44959 | 0.12.14 |
+-----------------------+---------+

dask
+-----------+---------+
|           | version |
+-----------+---------+
| client    | 0.17.2  |
| scheduler | 0.17.1  |
+-----------+---------+

distributed
+-----------------------+---------+
|                       | version |
+-----------------------+---------+
| client                | 1.21.6  |
| scheduler             | 1.21.1  |
| tcp://10.16.0.7:45136 | 1.21.1  |
| tcp://10.16.1.8:36913 | 1.21.1  |
| tcp://10.16.2.5:44959 | 1.21.1  |
+-----------------------+---------+

msgpack
+-----------------------+---------+
|                       | version |
+-----------------------+---------+
| client                | 0.5.1   |
| scheduler             | 0.5.5   |
| tcp://10.16.0.7:45136 | 0.5.5   |
| tcp://10.16.1.8:36913 | 0.5.5   |
| tcp://10.16.2.5:44959 | 0.5.5   |
+-----------------------+---------+

numpy
+-----------------------+---------+
|                       | version |
+-----------------------+---------+
| client                | 1.14.0  |
| scheduler             | 1.14.1  |
| tcp://10.16.0.7:45136 | 1.14.2  |
| tcp://10.16.1.8:36913 | 1.14.2  |
| tcp://10.16.2.5:44959 | 1.14.2  |
+-----------------------+---------+

And I'm currently having an error where is just takes an infinite amount of time to simply call df.head() from the cluster

[mrocklin]

Do those always reflect the latest version? Because when I use helm install stable/dask, and after updating all of my client packages, this is the check versions now:

No, they reflect the version that is in the helm chart. You have two options:

1. Build and push your own docker image. Presumably small changes to https://github.com/dask/dask-docker can get you where you want to be. You'll likely end up needing to do this at some point anyway
2. Set your local client version of Dask to be the same as what is in your image.

[BrendanMartin]

Why is it that in that dask-docker repo, the Dockerfile has dask==0.17.2 and distributed==1.21.6, but when I actually install via the dask helm chart, my cluster has dask 0.17.1 and distributed 1.21.1?

[BrendanMartin]

I went with building my own image, so I cloned the dask-docker repo and just removed the versions from the "RUN conda install ..." in the Dockerfile.

I built the image and pushed it to Docker Cloud, then put my repo's name in the config.yaml when installing stable/dask with helm.
config.yml:

scheduler:
  image:
    repository: "brendanmartin/dask"
    tag: "latest"

worker:
  image:
    repository: "brendanmartin/dask"
    tag: "latest"
  env:
    - name: EXTRA_CONDA_PACKAGES
      value: gcsfs>=0.1.0 dask-searchcv -c conda-forge
    - name: EXTRA_PIP_PACKAGES
      value: gcsfs>=0.1.0 dask-searchcv --upgrade

jupyter:
  enabled: false

Installing stable/dask with -f config.yaml, my pods error out. All of them have status "CrashLoopBackOff".

Do you have any idea what's going on here?

[mrocklin]

When I tried this locally I got errors like the following:

wiggly-clownfish-dask-scheduler-767f875986-9hlrk   0/1       ImagePullBackOff   0          15s
wiggly-clownfish-dask-worker-56bbb6b66d-2s8cx      0/1       ImagePullBackOff   0          15s
wiggly-clownfish-dask-worker-56bbb6b66d-qsbj5      0/1       ErrImagePull       0          15s
wiggly-clownfish-dask-worker-56bbb6b66d-rvws5      0/1       ErrImagePull       0          15s

I notice that this 404's https://hub.docker.com/r/brendanmartin/dask/

We pushed daskdev/dask:0.17.4 and daskdev/dask-notebook:0.17.4 to docker hub with the recent release. You could also try those.