index.rst

Jug: A Task-Based Parallelization Framework

What is Jug?

Jug allows you to write code that is broken up into tasks and run different tasks on different processors.

It currently has two backends. The first uses the filesystem to communicate between processes and works correctly over NFS, so you can coordinate processes on different machines. The second is based on redis so the processes only need the capability to connect to a common redis server.

Jug also takes care of saving all the intermediate results to the backend in a way that allows them to be retrieved later.

Examples

Short Example

Here is a one minute example. Save the following to a file called primes.py:

../../examples/primes/primes.py

Of course, this is only for didactical purposes, normally you would use a better method. Similarly, the sleep function is so that it does not run too fast.

Now type jug status primes.py to get:

Task name                     Waiting       Ready    Finished     Running
-------------------------------------------------------------------------
primes.is_prime                     0          99           0           0
.........................................................................
Total:                              0          99           0           0

This tells you that you have 99 tasks called primes.is_prime ready to run. So run jug execute primes.py &. You can even run multiple instances in the background (if you have multiple cores, for example). After starting 4 instances and waiting a few seconds, you can check the status again (with jug status primes.py):

Task name                     Waiting       Ready    Finished     Running
-------------------------------------------------------------------------
primes.is_prime                     0          63          32           4
.........................................................................
Total:                              0          63          32           4

Now you have 32 tasks finished, 4 running, and 63 still ready. Eventually, they will all finish and you can inspect the results with jug shell primes.py. This will give you an ipython shell. The primes100 variable is available, but it is an ugly list of jug.Task objects. To get the actual value, you call the value function:

In [1]: primes100 = value(primes100)

In [2]: primes100[:10]
Out[2]: [True, True, False, True, False, True, False, False, False, True]

More Examples

There is a worked out example in the tutorial, and another, fully functioning in the examples/ directory.

Links

• Mailing list
• Github
• Freshmeat
• Documentation
• Homepage

How do I get Jug?

The simplest is using pip:

pip install jug

You can either get the git repository at

http://github.com/luispedro/jug

Or download the package from PyPI

Testimonials

"I've been using jug with great success to distribute the running of a reasonably large set of parameter combinations" - Andreas Longva

Documentation Contents

tutorial.rst decrypt-example.rst text-example.rst segmentation-example.rst subcommands.rst status.rst shell.rst types.rst tasks.rst tasklets.rst invalidate.rst idioms.rst barrier.rst compound.rst utilities.rst mapreduce.rst backends.rst configuration.rst bash.rst faq.rst why.rst writeabackend.rst magic.rst api.rst history.rst

What do I need to run Jug?

It is a Python only package. Jug is continuously tested with Python 2.6 and up (including Python 3.3 and up).

How does it work?

Read the tutorial.

What's the status of the project?

Since version 1.0, jug should be considered stable.

Indices and tables

• genindex
• modindex
• search