SAGA-based Pilot-Job Implementation for Compute and Data
Python CSS JavaScript Shell
Latest commit d6cf865 Jul 28, 2014 @drelu drelu Revert "bootstrap to BJ2"
This reverts commit 71c6426.
Permalink
Failed to load latest commit information.
api Added wait() function to bigjob class Feb 2, 2012
bigjob Revert "bootstrap to BJ2" Jul 27, 2014
bigjob_dynamic Remove appwriting and pilotdata, these should not be front-facing pag… Dec 5, 2013
bootstrap Revert "bootstrap to BJ2" Jul 27, 2014
cli further refinement of API documentation Jul 7, 2012
coordination Merge branch 'master' into develop Oct 26, 2013
doc Remove all references to the gw68 password in code and examples Mar 21, 2013
docs Fix indices Dec 5, 2013
examples Revert "bootstrap to BJ2" Jul 27, 2014
pilot fixed examples so that they do not contain compile errors Jan 22, 2014
scripts work on path handling for irods Dec 14, 2012
tests optimizing performance of ComputeDataService Jul 31, 2013
util fix get_details() Jul 9, 2013
.gitignore Further work on Google Storage integration for PD Aug 11, 2012
CHANGES updated VERSION for release Jan 15, 2012
LICENSE.md Added License (so people are forced now to attribution) Jul 29, 2012
MANIFEST.in added VERSION to package Jan 23, 2014
Makefile added 'clean' target Jun 28, 2013
README-CONDOR.md separate README for Condor Jan 31, 2012
README.md Merge branch 'develop-prod' Jul 27, 2013
bigjob.conf updated bootstrapped version of BJ Aug 11, 2013
bigjob.sublime-project alt. PyPi project to minimize potential damage Jun 28, 2013
bigjob_agent.conf Make executor threadpool size configurable Jul 15, 2013
ez_setup.py use setuptools 2.1 for packaging bj Jan 24, 2014
setup.py force setuptools 2.x switch Jan 25, 2014

README.md

BigJob

The SAGA BigJob framework is a SAGA-based pilot job implementation. The Simple API for Grid Applications (SAGA) is a high-level, easy-to-use API for accessing distributed resources. SAGA BigJob supports a wide range of application types, and is usable over a broad range of infrastructures, i.e., it is general-purpose, extensible and interoperable. Unlike other common pilot job systems SAGA BigJob (i) natively supports MPI job and (ii) works on a variety of back-end systems, generally reflecting the advantage of using a SAGA-based approach.

Web Page & Mailing List

Web page: http://saga-project.github.io/BigJob/

Mailing list:

bigjob-user@googlegroups.com

bigjob-devel@googlegroups.com

Installation

$ curl https://raw.github.com/saga-project/BigJob/master/bootstrap/bigjob-bootstrap.py -o bigjob-bootstrap.py
$ python bigjob-bootstrap.py $HOME/.bigjob/python/
$ . $HOME/.bigjob/python/bin/activate

See https://github.com/saga-project/BigJob/wiki/ for further information. The Wiki includes among many things information on how to setup BigJob on various machines, e.g. on XSEDE (Ranger, Kraken) or FutureGrid.

Configuration of Redis Coordination Backend

BigJob requires Redis for communication between BigJob manager/agent. Use the following steps to install Redis.

Mac Homebrew:

brew install redis redis-tools 

Linux: Download Redis http://redis.io/download Currently only tested with Redis 2.2.12

$ tar -xzvf redis-2.2.12.tar.gz
$ cd redis-2.2.12
$ make
$ make install (if root)

Start Redis (redis-server executable is located in src/ dir):

cd src
./redis-server

Examples:

https://github.com/saga-project/BigJob/tree/master/examples/pilot-api

Please modify COORDINATION_URL on top of example to the correct Redis backend!

Development & Packaging

Requirements:

Building PyPi package

python setup.py build

Upload to PyPi

python setup.py sdist upload