dask-cuda on colab: ImportError: cannot import name 'SpecCluster'

[garanews]

There is anyone that was able to install dask-cuda and use it on colab notebook?
When I try to import LocalCUDACluster I am getting an error:

/usr/local/lib/python3.6/site-packages/dask_cuda/dgx.py in ()
1 import os
2
---->3 from dask.distributed import Nanny, SpecCluster, Scheduler
4 from distributed.worker import TOTAL_MEMORY
5

ImportError: cannot import name 'SpecCluster'

Do you have any idea?

[mrocklin]

It looks like your version of distributed is quite old. You might try pip install distributed --upgrade or conda install distributed.

How did you install dask-cuda?

[garanews]

I did some other try, this time without conda, just with pip.
I upgraded distributed (because default one is v 1.15) to 2.4 then pip install dask-cuda (so v 0.6)
I have this error:

I tried also to install dask-cuda from git (so v 0.10) but same issue:

I tried with distributed from git but no success, same issue.
Version with pip list are:
dask-cuda 0.10.0a0+14.ga05de10
distributed 2.4.0+5.g4bfe42d8

[pentschev]

RAPIDS no longer supports pip and isn't releasing packages there, the last one is 0.6.0, which is very old and shouldn't be used.

The TOTAL_MEMORY import error has been fixed last week in #131. I would double check that the versions you installed are indeed the ones being used. I just installed both dask-cuda and distributed from git and it works locally for me:

In [1]: import distributed, dask_cuda                                                                                                                                                                       

In [2]: distributed.__version__                                                                                                                                                                             
Out[2]: '2.4.0+5.gda70175.dirty'

In [3]: dask_cuda.__version__                                                                                                                                                                               
Out[3]: 'v0.10.0a+14.ga05de10'

[garanews]

I have it running locally too, what I would to achieve is implement it over colab, starting from your notebook sample present on rapids website:

but it seems that in this way when calling dask_cuda it uses distributed 1.15 that of course is not working, try to conda update it but says it is already at version 2.4.
with pip3 list I see it at 1.15, so tried to pip3 upgrade distributed, it will upgrade to 2.4 but ask for tornado 6 that is not compatible with env so ask to install ipykernel and restart env, so I'm like in a loop :)
if you have some time to try to load it and show over colab, it would be great, I think also for other people...
(fyi cudf and cuml are imported without issue from colab, just dask_cuda at the moment I am not able to load...)

[mrocklin]

@raydouglass any objection to us publishing dask-cuda on PyPI? If you don't want to handle it (reasonable) are you comfortable expanding permissions to other folks on PyPI?

[garanews]

@mrocklin did you have a chance to try load dask_cuda in your rapids-colab.ipynb ?
Personally pypi or conda doesn't matter, I would achieve have dask_cuda running over colab :)

[mrocklin]

I haven't tried this personally, no.

…

On Fri, Sep 20, 2019 at 2:16 AM garanews ***@***.***> wrote: @mrocklin <https://github.com/mrocklin> did you have a chance to try load dask_cuda in your rapids-colab.ipynb ? Personally pypi or conda doesn't matter, I would achieve have dask_cuda running over colab :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#132?email_source=notifications&email_token=AACKZTGGEPKT4O4H2JWZFUDQKR2DPA5CNFSM4IXLLOJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7F2DQQ#issuecomment-533438914>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACKZTASSYLNEJOH2WCERKDQKR2DPANCNFSM4IXLLOJA> .

[pentschev]

This seems to have stalled, perhaps it has been already solved in newer versions of dask-cuda and dask in general. I'm therefore closing it, but feel free to reopen if that's still an issue.