0.10.1 sge #152

Merged
15 commits merged into from Sep 24, 2010

5 participants

@satra

ssh now supports sending local environment variables over with the sshx.sh script. this may or may not be useful in all situations, but one thing we should try to determine while testing is whether this generates any security loopholes.

Satrajit Ghosh and others added some commits Jul 16, 2010
Satrajit Ghosh initial sge compatibility attemp b222b5c
Satrajit Ghosh firt pass to support sge e9f4279
Satrajit Ghosh updated sge to launch multiple engines - start cleanly but does not s…
…hutdown cleanly
70725a5
@jtriley jtriley update SGEEngineSet to use SGE job arrays
SGE job arrays allow one job id to be associated with a set of processes on
an SGE cluster.

modified SGEEngineSet to be a subclass PBSEngineSet

ipcluster will now generate a default SGE job script if --sge-script is
not provided. most folks should ignore the --sge-script option unless they
know they need it.

if --sge-script is passed, check that the script exists and that the user has
defined a "#$ -t" setting within the script. if not, add the setting
for them by copying the script to a temp file and launching the job
array using the modified temp file.

ipengines terminate cleanly now when the ipcluster command exits.

i think we still need to handle furl files when engines are assigned to different
hosts via SGE. without using ssh or nfs, the only other way is to put
the contents of the furl in the job script before submission but this is
less secure. need to discuss this.
225cd96
@jtriley jtriley update PBSEngineSet to use job arrays
refactored base class BatchEngineSet to handle job arrays. PBS and SGE
engine sets are much simpler now.
1edb987
@jtriley jtriley add experimental LSF support (needs testing) a0e2363
@jtriley jtriley add main_lsf for experimental LSF support fe180bb
@jtriley jtriley add --queue to ipcluster's SGE/PBS/LSF subcmds 7952c77
@mbrucher mbrucher Fixes for LSF bfb3a62
@jtriley jtriley force /bin/sh in default SGE/PBS/LFS templates 11c0776
@jtriley jtriley add bsub wrapper for LSFEngineSet
LSF requires that job scripts be submitted to bsub through stdin.
twisted.internet.utils.getProcessOutput does not support shell
redirection so the bsub wrapper takes care of doing this.
09567ca
@jtriley jtriley update ipcluster docs w/ latest PBS/SGE/LSF changes 1505a87
@satra satra ENH: ssh now allows sending all current environment args to remote ip…
…engine
1362bd4
@satra satra DOC: updated parallel engine docs to include default script information 45e3e05
Satrajit Ghosh FIX: updated ssh environment copying to escape special characters for…
… posix compatibility
4beba07
@fperez
IPython member

Many thanks for this, guys!

Merged and closed with f780f2e.

@jedludlow jedludlow pushed a commit that referenced this pull request Dec 19, 2012
@fperez fperez Merge branch '0.10.1-sge' of http://github.com/satra/ipython into sat…
…ra-0.10.1-sge

This branch adds Sun Grid Engine support to the ipcluster startup
system.

Credits: Justin Riley and Satra Ghosh @ MIT, Matthew Brucher.

Closes gh-152 (pull request)
f780f2e
@minrk minrk added this to the old milestone Apr 10, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment