Helm dask notes seem to have a typo in setting the environment variable

[twiecki]

I followed the guide at: http://docs.dask.org/en/latest/setup/kubernetes-helm.html

Running helm install stable/dask instructs me to run:

to set the right env variables. Note that the variable is called DASK_SCHEDULER while dask seems to look for DASK_SCHEDULER_ADDRESS, so

was not working for me unless I fixed that.

[twiecki]

Actually that doesn't seem to be right either as it requires tcp:// in front it seems. Is that env variable supposed to be set automatically? Because it was not.

[mrocklin]

Yup, that looks like a genuine issue. We used to use the `DASK_SCHEDULER` environment variable, but changed this to `DASK_SCHEDULER_ADDRESS` at some point in the last year. Changing this upstream in the helm chart would be good. I plan to update that chart sometime in the next couple weeks (after a release soonish) and so can probably handle it then. If anyone wants to beat me to it that would also be welcome :)

…

On Tue, Nov 27, 2018 at 2:50 PM Thomas Wiecki ***@***.***> wrote: I followed the guide at: http://docs.dask.org/en/latest/setup/kubernetes-helm.html Running helm install stable/dask instructs me to run: [image: image] <https://user-images.githubusercontent.com/674200/49107365-cf762a00-f285-11e8-9924-d7b003af308b.png> to set the right env variables. Note that the variable is called DASK_SCHEDULER while dask seems to look for DASK_SCHEDULER_ADDRESS, so [image: image] <https://user-images.githubusercontent.com/674200/49107414-f3d20680-f285-11e8-944f-234b80b7b652.png> was not working for me unless I fixed that. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4253>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszCA8oKchHctd7E0CJsT8iOCCm2Vqks5uzZdygaJpZM4Y2N2l> .

[mrocklin]

Thanks for reporting @twiecki

…

On Tue, Nov 27, 2018 at 2:55 PM Matthew Rocklin ***@***.***> wrote: Yup, that looks like a genuine issue. We used to use the `DASK_SCHEDULER` environment variable, but changed this to `DASK_SCHEDULER_ADDRESS` at some point in the last year. Changing this upstream in the helm chart would be good. I plan to update that chart sometime in the next couple weeks (after a release soonish) and so can probably handle it then. If anyone wants to beat me to it that would also be welcome :) On Tue, Nov 27, 2018 at 2:50 PM Thomas Wiecki ***@***.***> wrote: > I followed the guide at: > http://docs.dask.org/en/latest/setup/kubernetes-helm.html > > Running helm install stable/dask instructs me to run: > [image: image] > <https://user-images.githubusercontent.com/674200/49107365-cf762a00-f285-11e8-9924-d7b003af308b.png> > > to set the right env variables. Note that the variable is called > DASK_SCHEDULER while dask seems to look for DASK_SCHEDULER_ADDRESS, so > [image: image] > <https://user-images.githubusercontent.com/674200/49107414-f3d20680-f285-11e8-944f-234b80b7b652.png> > was not working for me unless I fixed that. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#4253>, or mute the thread > <https://github.com/notifications/unsubscribe-auth/AASszCA8oKchHctd7E0CJsT8iOCCm2Vqks5uzZdygaJpZM4Y2N2l> > . >

[mrocklin]

The tcp:// is desired, but will be prepended by default if not present.

…

On Tue, Nov 27, 2018 at 2:56 PM Thomas Wiecki ***@***.***> wrote: Actually that doesn't seem to be right either as it requires tcp:// in front it seems. Is that env variable supposed to be set automatically? Because it was not. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4253 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszL3UjYnWv4YnmbI_QmnHGzFv1XVEks5uzZjYgaJpZM4Y2N2l> .

[twiecki]

Actually I'm not quite sure I'm doing it right:
You can create a notebook and create a Dask client from there. The DASK_SCHEDULER_ADDRESS environment variable has been populated with the address of the Dask scheduler. This is available in Python in the config dictionary.

I ran ipython locally instead, it works fine when I connect to the jupyter lab server and run it there. Or is it supposed to work locally as well?

[mrocklin]

You need to provide the Client object with a network address that connects to the dask-scheduler pod. Within Kubernetes the bald-eel-scheduler:8786 address should work. Outside of Kubernetes you'll need to find an address that is externally visible. This should be accessible with the helm or kubectl CLI, though I admit that the exact command escapes me at the moment.

[twiecki]

if I run export DASK_SCHEDULER_ADDRESS=tcp://$DASK_SCHEDULER:8786 it works fine with just instantiating Client, although then running something fails locally then with:

In [3]: >>> def square(x):
   ...:         return x ** 2
   ...:
   ...: >>> def neg(x):
   ...:         return -x
   ...:
   ...: >>> A = client.map(square, range(10))
   ...: >>> B = client.map(neg, A)
   ...: >>> total = client.submit(sum, B)
   ...: >>> total.result()
   ...:
   ...:
---------------------------------------------------------------------------
CancelledError                            Traceback (most recent call last)
<ipython-input-3-0b952c058b1d> in <module>()
      8 B = client.map(neg, A)
      9 total = client.submit(sum, B)
---> 10 total.result()

~/anaconda3/lib/python3.6/site-packages/distributed/client.py in result(self, timeout)
    195             six.reraise(*result)
    196         elif self.status == 'cancelled':
--> 197             raise result
    198         else:
    199             return result

CancelledError: sum-5c00f435c56b61583526e7a19aa3a2bd

[mrocklin]

You should check worker logs. But my guess is that you have mismatched versions between your local client library and the version used in the helm chart. You might consider running client.get_versions(check=True). If I'm right then you'll get some exception.

[twiecki]

TypeError: versions() got an unexpected keyword argument 'packages', I guess that means yes? :)

[mrocklin]

Yes. If you're within a decent range of versions then you'll get a nicely printed out version of which packages are out of sync. If you're mismatched enough then you get that :/

The docker image in the stable helm chart is pretty old. You might consider bumping it up with a small config file. Something like the following:

scheduler:
  image: "daskdev/dask"
  tag: 0.19.4

worker:
  image: "daskdev/dask"
  tag: 0.19.4

notebook:
  image: "daskdev/dask-notebook"
  tag: 0.19.4

Then helm upgrade bald-eagle -f my-config.yaml

It looks like there isn't a docker image newer than that up on docker hub.

[mrocklin]

I've tagged a 0.20.2 on docker hub. Should be built and up in a while.

[martindurant]

If you're mismatched enough then you get that :/

Presumably check_versions should be resilient to this (for the future) and give a sensible error if it failed to fetch information successfully.

[mrocklin]

Yes, that would be ideal. Anyone want to submit a PR fixing that?

…

On Tue, Nov 27, 2018 at 3:50 PM Martin Durant ***@***.***> wrote: If you're mismatched enough then you get that :/ Presumably check_versions should be resilient to this (for the future) and give a sensible error if it failed to fetch information successfully. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4253 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszCE2XhiYNJmAjlwRyCR9czbQePRYks5uzaWmgaJpZM4Y2N2l> .

[twiecki]

If I run helm repo update and helm install stable/dask should I get dask containers with 0.20.2? Because the version still fails.

[mrocklin]

No, the helm stable channel moves very slowly. You need to modify your configuration by providing a config file like in the post above that specifies new docker images. Then you can install or upgrade your deployment with those modified values.

…

On Wed, Nov 28, 2018 at 11:17 AM Thomas Wiecki ***@***.***> wrote: If I run helm repo update and helm install stable/dask should I get dask containers with 0.20.2? Because the version still fails. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4253 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszO8Thz6w-1J6lsBHUW-CdgwuVWnDks5uzrchgaJpZM4Y2N2l> .

[mrocklin]

There are a bunch of default values in the chart listed here: https://github.com/helm/charts/blob/master/stable/dask/values.yaml You can override those values by providing your own yaml file with a subset of the values there. In our case we want to update the docker image tags to a newer version.

…

On Wed, Nov 28, 2018 at 11:25 AM Matthew Rocklin ***@***.***> wrote: No, the helm stable channel moves very slowly. You need to modify your configuration by providing a config file like in the post above that specifies new docker images. Then you can install or upgrade your deployment with those modified values. On Wed, Nov 28, 2018 at 11:17 AM Thomas Wiecki ***@***.***> wrote: > If I run helm repo update and helm install stable/dask should I get dask > containers with 0.20.2? Because the version still fails. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#4253 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/AASszO8Thz6w-1J6lsBHUW-CdgwuVWnDks5uzrchgaJpZM4Y2N2l> > . >