Embarrasingly distributed parallel loop on clusters of computers
The goal of this library is to ease the creation, launch and management of embarassingly parallel jobs on supercomputer with schedulers such as SLURM. Some basic primitives (e.g. pure python NO-SQL database) are also provided to work in distributed memory architecture.
Aims: simple, pure python
If you want to parallelize your python jobs in shared memory architecture, I advise you to use joblib.
Getting the latest code
To get the latest code using git, simply type:
git clone git://github.com/clusterlib/clusterlib.git
If you don't have git installed, you can download a zip or tarball of the latest code: https://github.com/clusterlib/clusterlib/archive/master.zip
As any Python packages, to install clusterlib, simply do:
python setup.py install
in the source code directory. You can also use
pip install clusterlib
How to contribute?
To contribute to clusterlib, first create a github account. Then you can fork the clusterlib repository. Once this is done, you can make clone of your fork, make your changes and whenever you are happy, send us a pull request to the main repository.
Running the test suite
To run the test suite, you need nosetests and the coverage modules. Run the test suite using:
from the root of the project.
For making the documentation, Sphinx==1.2.2 and sphinx-bootstrap-theme==0.4.0 are needed. Then, you can do
How to make a release
What follows is only for maintaners:
Create a branch for the 0.X.Y version if necessary.
Update What's new in the 0.X.Y version and master branch. Update version in the documentation at
Make point the stable branch to 0.X.Y.
Check the generated doc on read the doc for the stable and the 0.X.Y tag version.
Check release on pypi test using
python setup.py sdist upload -r pypitest
Upload the new release on pypi using
python setup.py sdist upload -r pypi