Nanny fails on OpenMPI+UCX

[jacobtomlinson]

In the documentation, you clearly mention that --nanny flag works only for some MPI implementations because of process forking. As far as I know, OpenMPI implementations with verbs support complains about process forking. But, OpenMPI with UCX support do not have this problem, it is quite happy to allow process forking. We have few MPI codes that do this and we did not find any problems with OpenMPI+UCX. Unfortunately, dask-mpi with --nanny flags still fails with this OpenMPI+UCX implementation. Are there any MPI implementations that you have tested will work for this setting? Thanks

Originally posted by @mahendrapaipuri in #72 (comment)

[kmpaul]

Hey, @jacobtomlinson: I only tested OpenMPI (without UCX) and MPICH. With the old version of MPICH, I was able to get the --nanny option to work, I think. But maybe things have changed and it doesn't work with any flavor of MPI. I probably need to make time to test across the spectrum of packages. I always felt that just testing against MPICH and OpenMPI would be enough.

[kmpaul]

@jacobtomlinson: I had a moment to look into this further. It actually comes down to spawned processes, and not forked processes. I created a simple mpi4py test of this with MPICH:

#!/usr/bin/env python
#
# This routine demonstrates that MPI processes CAN 'fork' other
# processes, but they cannot 'spawn' them.
#

import multiprocessing
import os

from mpi4py import MPI


def info(title, rank):
    print(f'{title}:')
    print(f'  mpi_rank: {rank}')
    print(f'  module name: {__name__}')
    print(f'  parent process: {os.getppid()}')
    print(f'  process id: {os.getpid()}', flush=True)


if __name__ == '__main__':
    comm = MPI.COMM_WORLD
    rank = comm.Get_rank()

    mp_type = 'spawn' # or 'fork'
    mp_context = multiprocessing.get_context(mp_type)

    info('main', rank)

    p = mp_context.Process(target=info, args=('proc', rank,))
    p.start()
    p.join()

If you name this script multiproc_mpi.py and run this script with mpirun -n 2 python multiproc_mpi.py, you will get an error from MPICH. However, if you change mp_type to 'fork', it will not error!

This suggests that if you set distributed.worker.multiprocessing-method to fork instead of its default (spawn), that dask-mpi should work with with the --nanny option. So, I tested it by adding:

distributed:
  worker:
    multiprocessing-method: fork

to the Dask Distributed config file (~/.config/dask/distributed.yml). ...And I found a bug in dask-mpi:

Error output from "mpirun -n 4 dask-mpi"

(dask-mpi-nanny) kpaul@cisl-brainerd: dask-mpi> mpirun -n 4 dask-mpi
distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO -   Scheduler at:  tcp://192.168.1.12:52715
distributed.scheduler - INFO -   dashboard at:                     :8787
Traceback (most recent call last):
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/bin/dask-mpi", line 11, in <module>
Traceback (most recent call last):
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/bin/dask-mpi", line 11, in <module>
Traceback (most recent call last):
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/bin/dask-mpi", line 11, in <module>
    sys.exit(go())
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/dask_mpi/cli.py", line 112, in go
    sys.exit(go())
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/dask_mpi/cli.py", line 112, in go
    sys.exit(go())
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/dask_mpi/cli.py", line 112, in go
    main()
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    main()
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    main()
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/click/core.py", line 1062, in main
    return self.main(*args, **kwargs)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/click/core.py", line 1062, in main
    return self.main(*args, **kwargs)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    rv = self.invoke(ctx)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    rv = self.invoke(ctx)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/dask_mpi/cli.py", line 107, in main
    return __callback(*args, **kwargs)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/dask_mpi/cli.py", line 107, in main
    return __callback(*args, **kwargs)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/dask_mpi/cli.py", line 107, in main
    asyncio.get_event_loop().run_until_complete(run_worker())
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    asyncio.get_event_loop().run_until_complete(run_worker())
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    asyncio.get_event_loop().run_until_complete(run_worker())
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/dask_mpi/cli.py", line 96, in run_worker
    return future.result()
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/dask_mpi/cli.py", line 96, in run_worker
    return future.result()
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/dask_mpi/cli.py", line 96, in run_worker
    async with WorkerType(
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/distributed/nanny.py", line 138, in __init__
    async with WorkerType(
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/distributed/nanny.py", line 138, in __init__
    async with WorkerType(
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/site-packages/distributed/nanny.py", line 138, in __init__
    os.makedirs(self.local_directory, exist_ok=True)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/os.py", line 225, in makedirs
    os.makedirs(self.local_directory, exist_ok=True)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/os.py", line 225, in makedirs
    os.makedirs(self.local_directory, exist_ok=True)
  File "/Users/kpaul/Software/miniconda3/envs/dask-mpi-nanny/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''
^C[mpiexec@cisl-brainerd] Sending Ctrl-C to processes as requested
[mpiexec@cisl-brainerd] Press Ctrl-C again to force abort

This traces back to here:

dask-mpi/dask_mpi/cli.py

Line 40 in b1638a4

"--local-directory", default="", type=str, help="Directory to place worker files"

The --local-directory option defaults to an empty str, which does not evaluate to True in nanny.py:

https://github.com/dask/distributed/blob/3754a6694f72207b69aaf661445aecf18fb5f0cb/distributed/nanny.py#L129-L134

Hence, the Nanny tries to create a directory without a name. If you fix this with:

@click.option(
    "--local-directory", default=None, type=str, help="Directory to place worker files"
)

Then mpirun -n 4 dask-mpi works as expected.

[kmpaul]

So, I think the solution to this is:

• A PR that fixes dask-mpi/cli.py to correctly set the default value of the --local-directory option
• New documentation for Dask-MPI that informs the user to set distributed.worker.multiprocessing-method to fork if they want to use the --nanny option (which is the default).

[kmpaul]

Maybe this also means that --no-nanny should be the default for dask-mpi?

[mahendrapaipuri]

@kmpaul, Excellent. Thanks for bugging it out. As soon as you have a PR setup, I can test the patch and give you some feedback.
I am my tests, even with --no-nanny option, dask-mpi seems to break when using with "real" applications with huge datasets involved. I got a ton of exceptions and errors from deep inside the distributed modules. I did not have to investigate it further. I will redo the tests with new patch and provide you some feedback.

[kmpaul]

@mahendrapaipuri: We'll see what we can find, but I'm sure it will be helpful to fix the immediate problems now. One step at a time. 😄

[kmpaul]

@mahendrapaipuri @jacobtomlinson: It's unclear to me if this issue has been addressed with the recent bugfix #77 as well as the recent PR merges (#73 and #75). @mahendrapaipuri, are you able to test with the main branch straight from GitHub?

[mahendrapaipuri]

@kmpaul Well, I finally managed to test the PR. So, I used the main branch from the GitHub and dask-mpi worked without any issues both on "toy" and "application" problems. I tested with both OpenMPI 4.1.1 + UCX and OpenMPI 3.1.3 + verbs and in both cases it worked. With OpenMPI 3.1.3 + verbs support, OpenMPI threw me some warnings about processing forking though (which is normal). Seems like these PRs fixed most of the problems we had before. I think if you could update the documentation as well, that would be really helpful!

Thanks @kmpaul for fixing the bugs!

[kmpaul]

@mahendrapaipuri: That's excellent news! When I discovered the --local-directory bug (#77), and found errors related to it in all of the open PRs, I was suspicious that it might be related...and at the very least was masking any real problems. I'm glad it's working for you!

Next step is to update the documentation and then get out a new release. Thanks for testing!

[mahendrapaipuri]

@kmpaul @jacobtomlinson When do you plan to have next release with these bug fixes? Thanks!

[kmpaul]

Well, good question. Looking at my schedule, I think I can have something out in a week.