Compmake is a non-obtrusive module that provides Makefile-like facilities (including parallel processing) for batch Python applications.
Python Makefile
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.circleci
docs Merge branch 'master' into env_master_writingtools Jun 14, 2015
src/compmake
.bumpversion.cfg
.gitignore
.travis.yml
LICENSE.txt
MANIFEST.in
Makefile
README.md
examples
pypackage.mk
requirements.txt
setup.py
shippable.yml

README.md

Compmake

Compmake is an unobstrusive Python library that provides Make--like facilities to a Python application, including:

  • Minimal effort job management and parallelization (multiple CPU on a single host, cluster computing using SGE, and experimental support for cloud computing using Multyvac).
  • Caching of temporary results: you can interrupt your program and restart it without losing what was already computed.
  • Failure tolerance: if a job fails, other jobs that do not depend on it continue to be executed.
  • A console for inspecting failures and partial completion, with familiar commands such as make, clean, etc.

Please see the manual at:

http://purl.org/censi/compmake-manual

Issues

Please report any problem using Github's issue tracker at

http://github.com/AndreaCensi/compmake/issues

Acknowledgements

Your tax dollars at work! Compmake's development was supported by the National Science Foundation in the National Robotics Initiative program under grant #1405259.

Screencasts

Some animations of the job graph

Robustness to job failure

Simple dynamic jobs

Dynamic jobs and recursive parallel executions

Changelog

  • v3.5 - Added why command --- compact error visualization. Removed color effects which might not be suitable for all console types.
  • v3.4.1 - Bug fix; experimental multyvac_sync
  • v3.4.0 - Changed the way that dynamic job IDs are generated to avoid a race condition in a corner case.
  • v3.3.7 - Bug fix.
  • v3.3.6 - Experimental Multyvac backend.