Skip to content
Go to file

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


- What is this?
- Requirements
- Licensing
- Files
- Installation and configuration
- Creating new jobs
- Bug / Feature requests

This is basically a job dispatcher. For inexplicable reasons it's called "upq".
Atm this project is hosted at

* Jobs
Jobs are written as modules that can be run with arguments through a command on
a UNIX/NET socket. Each module defines a "Job"-type. A queue is automatically
created for each Job-type. A per-Job-type configurable amount of worker threads
consume from each queue. Jobs run in their own thread. The standard Python Queue
was extended to make Jobs persistent in a MySQL DB to withstand (upq) server
crash/restart. After the server starts it looks in the DB to see if any jobs
have not finished, and restarts them.

* Notifications
There is a basic notification system that atm can send emails and log to syslog
if a task finishes un/successful. Configuration is per-UpqJob in upq.cfg in
section [job xyz] -> "notify_fail" and  "notify_success".

* Naming conventions
The server accepts commands of the form "my_command argument". If there is a
section "[job my_command]" in upq.cfg it will search and load a module called
"". It searches in the path configured in upq.cfg in section
[paths] variable "jobs_dir". In the file "" it expects to find a
class "My_command" (first letter of module name upper case) that is derived from
the "UpqJob" class. 
In section "[job my_command]" all variables are availe through
self.jobcfg['variablename'] inside a job.

Python >= 2.6 (< 3.0)
SQLAlchemy 0.6
python-daemon (

on ubuntu run:

apt-get install python-sqlalchemy python-mysqldb python-daemon python-imaging

At this moment all code is covered by the GPLv3, see file COPYING.

Future additions of Jobs and Tasks may be done by other developers and my have
different licensing. Please look at the top of each file for an author and
license note.

You can put all of this software wherever you like - probably something like
/usr/local/upq. The directories for UpqJobs and UpqTasks are configurable.

For Job persistence data is stored in a Database. Configuration is in section
"[db]". Just edit the section to configure the db and start up upq. Tables
are created automaticly.

Configuration in upq.cfg is pretty self explanatory. Important: UpqJobs will
only be loaded if they have a section in upq.cfg!
Atm changes in upq.cfg need a server restart to take effect.

The server does not need (and should not) be started as a privileged user.
Choice of user account is dependent on what it should be able to do on your
files. By itself it only needs write permissions on its socket ([paths]->socket)
and log file ([logging]->logfile).

The whole point of upq is to be able to easily write and schedule Jobs by just
deriving from a class and editing an ini-file.

To create a new job do:
* Create a file in the directory [paths]->jobs_dir.
* must contain a "class Job_name(UpqJob)" deriving from UpqJob.
* Create a section "[job job_name]" in upq.cfg. The section should define
variables "enabled = <bool>", "concurrent = <int>",
"notify_fail = [syslog xyz | mail <email address list>]",
"notify_success = [syslog xyz | mail <email address list>]".
* If you wish to call another UpqJob from within an UpqJob and use its
<job>.result, use <job>.finished.wait() to block the calling UpqJob until
<job>.run() finishes. (The queue worker thread will release the lock.)

After restarting the server it is possible to start the new Job. Testing can be
performed with "" like this:
$ ./upq-cli "verify_local_file fid:50"

COPYING                 GNU General Public License Version 3              Python Queue class with persistent state in DB                  logging        finds and loads job and task modules               admin notification system          parse upq.cfg               Job classes derive from this
upq.cfg                 [ini]-style configuration file              cmd line client for testing purposes                DB tool class, all DB code is in this file                  main(), invoke this with "$ ./"         queue management            socket server & request handler code
jobs/                   UpqJob modules reside inside this directory, deployment
                        location is configurable in upq.cfg, for description
                        of files in this directory look into the header of
                        the file


Please report to


runs jobs at




No releases published


No packages published
You can’t perform that action at this time.