Underministic behavior of addprocs() of SSHManager

[JonnyKong]

Given an array of node addresses as input, this function returns an array of launched worker PIDs. However, the returned pids do not necesssarily match the order of input addresses.

For example, the outcome of (p1, p2) = addprocs([machine1, machine2]) may be p1 running on machine2 and p2 running on machine1, or vice versa.

The cause of such underministic behavior is that launch(manager::SSHManager, ...) launches workers in parallel. Upon launching each worker, the pid of that worker will be pushed to launched, where no synchronization / ordering is performed:

Distributed.jl/src/managers.jl

Lines 177 to 185 in fd9d120

	@sync for (i, (machine, cnt)) in enumerate(manager.machines)
	let machine=machine, cnt=cnt
	@async try
	launch_on_machine(manager, $machine, $cnt, params, launched, launch_ntfy)
	catch e
	print(stderr, "exception launching on machine $(machine) : $(e)\n")
	end
	end
	end

While this is not a bug, this undeterministic behavior seems counter-intuitive and is error-prone.

[JonnyKong]

Reopening this issue (was closed by mistake).