Can the scheduler info be broadcast at startup instead of using scheduler file?

[rcthomas]

If I hit the end of a job allocation, or I OOM my cluster, or the cluster is shut down abruptly, an old scheduler.json file can be found by the workers on my next try. I could always be sure to delete this file before I launch dask-mpi but why not just pass the info over the wire?

Sound handy? We'd probably get PR that in before I figured out my other MPI cluster question...

[mrocklin]

I think that using MPI to bootstrap this sort of information sounds like a good idea.

…

On Mon, Dec 9, 2019 at 2:20 PM R. C. Thomas ***@***.***> wrote: If I hit the end of a job allocation, or I OOM my cluster, or the cluster is shut down abruptly, an old scheduler.json file can be found by the workers on my next try. I could always be sure to delete this file before I launch dask-mpi but why not just pass the info over the wire? Sound handy? We'd probably get PR that in before I figured out my other MPI cluster question... — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#45?email_source=notifications&email_token=AACKZTGE2ALP3OUDLNBAWLLQX3ADDA5CNFSM4JYTQSZ2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H7JESWA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTGMBHNRJNGDYCPJHPDQX3ADDANCNFSM4JYTQSZQ> .

[kmpaul]

@rcthomas This is what is done in Dask-MPI. The scheduler is started, the address is broadcast to the workers.

[kmpaul]

@rcthomas Look here:

dask-mpi/dask_mpi/core.py

Lines 57 to 75 in de00068

	if rank == 0:
	async def run_scheduler():
	async with Scheduler(
	interface=interface,
	protocol=protocol,
	dashboard_address=dashboard_address,
	) as scheduler:
	comm.bcast(scheduler.address, root=0)
	comm.Barrier()
	await scheduler.finished()
	asyncio.get_event_loop().run_until_complete(run_scheduler())
	sys.exit()
	else:
	scheduler_address = comm.bcast(None, root=0)
	dask.config.set(scheduler_address=scheduler_address)
	comm.Barrier()

[rcthomas]

Thanks, I inferred something from the behavior I was seeing that actually looking at the code would have made obvious. OK. So something's wrong with what I'm doing; we'll investigate further as to how this is happening.