Add a nonblocking addprocs_sge (and others)

[cako]

Currently the use of addprocs_sge (and the others) is blocking. It must obtain the node before it returns. For example, applying @async to a command which calls addprocs_sge is useless.

I would like to know if is it possible to add a non-blocking version of addprocs_sge. I am happy to help with the coding but am not overly familiar with the addprocs structure.

I suggested a workaround in a question on StackOverflow but am not sure my approach to the issue is recommended.

[bjarthur]

if your SGE cluster supports qrsh, you might try addprocs_qrsh instead. IIRC, that does not block.

[cako]

Thanks @bjarthur, this works for me. Unfortunately addprocs_qrsh cannot take res_list like the others, but that should be easy enough to change. If I have time in the next few days I will try to sort it out.

I should add that since addprocs_qrsh creates an ssh tunnel to the worker, it is wise to disable .bash_profile on the worker. Otherwise commands defined inside it can mess up the communication between master and worker.

EDIT: Added pull request #82 to include res_list support in addprocs_qrsh

[cako]

Actually, upon trying this with more workers, it still does not submit the jobs asynchronously. Each processor must be added before the next job can be queued in the gist I linked. You can test it by running the script with, say 10 jobs, and executing watch qstat in another window. You will notice that new jobs are only submitted after the previous started running. The only way in which I've managed to remove that restriction is by substituting

id = addprocs_qrsh(1)

by

qrsh = QRSHManager(1, "");
Distributed.cluster_mgmt_from_master_check()
id = Distributed.addprocs_locked(qrsh; qsub_env="")

In this case, all jobs will be queued at the same time. For some reason SGEManager is a bit more stable for me when running many jobs, so for now I am sticking to that.

[juliohm]

@cako it is a nice feature to have a non-blocking addprocs_*. We are trying to get this package back in shape. Please submit a PR if it is not yet available. I am closing this issue as too old. Feel free to add another one or the PR directly.

[oameye]

Has a non-blocking addprocs ever been added? Is there a solution?