uri_from_host_port moved

[kbarros]

dask_mpi/common.py contains the line:

from distributed.cli.utils import uri_from_host_port

However, it appears that the uri_from_host_port function has moved to distributed/comm/addressing.py.

To resolve the compile error, I think the above import statement needs to be changed to:

from distributed.comm.addressing import uri_from_host_port

[andersy005]

@kbarros, thank you for reporting this. There's a work in progress PR #32 to fix this...

@kmpaul, @muammar: it might be a good time to revisit #32.

The latest releases of distributed and dask dropped Python 2.7 support. So, we may need to drop it as well.

[kmpaul]

Yes. We should drop Python 2.7 support.

Do we need backwards compatibility with dask <1.28.1?

[andersy005]

Do we need backwards compatibility with dask <1.28.1?

It's worth looking into what pieces were moved around in distributed >= 2.0 and dask >= 2.0 to see whether or not it would require major efforts in order to keep the backwards compatibility.

@mrocklin, I am interested to hear your thoughts on this.

[mrocklin]

Personally I'm in favor of moving all of the deployment solutions that are comfortable with dropping Python 2 up to Dask 2.0.

[kmpaul]

Sweet! We'll get to work on updating this.

@andersy005 Do you feel comfortable taking a stab at this? Remove Python 2 support? Make the necessary changes to get dask-mpi working with dask 2.0? Bump the version to 2.0?

[mrocklin]

For Python 2 support please remember to add the python_requires=">=3.5" keyword to setup.py and, when building, drop the --universal flag.

[kmpaul]

We will. Thanks, @mrocklin!

[kbarros]

I left a comment on #32 -- I can't get dask-mpi to work even with this fix.

[andersy005]

@kbarros, #36 addresses this issue. Feel free to try it at your earliest convenience and please let us know how it goes.

[kbarros]

@andersy005 thanks for working on this! I tried both
mpirun -np 4 python dask_mpi/tests/core_basic.py
and
pytest dask_mpi
and here is the result:
https://gist.githubusercontent.com/kbarros/732a1aa968daf559b2f48f9e4e0056d1/raw/11620c2214014a5384e30e7a04df773511711872/gistfile1.txt

With the new fixes it doesn't hang like before, but it does return some failures. Perhaps there is something in my python package installations is wrong, but I'm not sure how to debug that. mpi4py demo scripts seemed to work OK when I tried them. I'm using openmpi 4.0.0 built for gcc 6.4.0.

[andersy005]

@kbarros, Dask-mpi CI is using the following versions:

mpi                       1.0                     openmpi    conda-forge
mpi4py                    3.0.1            py36hd955b32_0    conda-forge
openmpi                   3.1.4                hc99cbb1_0    conda-forge

I will try to reproduce the issue with openmpi==4.0.0 and will let you know what I find.

[kbarros]

@andersy005 thanks for the info. Previously I was using mpi4py installed from pip, which used the system MPI. Now I switched to mpi4py installed from conda, which uses the mpich package:

mpi                       1.0                       mpich  
mpi4py                    2.0.0                    py36_2  
mpich2                    1.4.1p1                       0  

It seems like I get the same error in either case. Not sure how to investigate further.

[andersy005]

Can you try using a more recent release of mpi4py, and see if it makes any difference?

mpi                       1.0                       mpich    conda-forge
mpi4py                    3.0.1            py36hf046da1_0    conda-forge
mpich                     3.2.1             hc99cbb1_1012    conda-forge