From 8097bc80be2b58513586b960dfcaf1b88f37c300 Mon Sep 17 00:00:00 2001 From: Pete Wildsmith Date: Thu, 19 Mar 2015 15:27:37 +0000 Subject: [PATCH 1/4] update readme --- README.md | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 50 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 6b6cb87..1f9bef9 100644 --- a/README.md +++ b/README.md @@ -8,13 +8,52 @@ Asynchronous tasks are run via a *job queue*. This system is designed to support This is not yet production-ready, and the API is liable to change. Use at your own risk. -### Terminology +## Getting Started -#### Job +### Describe your job + +In e.g. project.common.jobs: + +```python +import time + +def my_task(job): + logger.info("Working hard...") + time.sleep(10) + logger.info("Job's done!") +``` + +### Set up your job + +In project.settings: + +```python +JOBS = { + 'my_job': ['project.common.jobs.my_task'], +} +``` + +### Start the worker + +In another terminal: + +`python manage.py worker` + +### Create a job + +Using the name you configured for your job in your settings, create an instance of Job. + +```python +Job.objects.create(name='my_job') +``` + +## Terminology + +### Job The top-level abstraction of a standalone piece of work. Jobs are stored in the database (ie they are represented as Django model instances). -#### Task +### Task Jobs are processed to completion by *tasks*. These are simply Python functions, which must take a single argument - the `Job` instance being processed. A single job will often require processing by more than one task to be completed fully. Creating the task functions is the responsibility of the developer. For example: @@ -22,7 +61,7 @@ Jobs are processed to completion by *tasks*. These are simply Python functions, logger.info("Doing some hard work") do_some_hard_work() -#### Workspace +### Workspace The *workspace* is an area that tasks within a single job can use to communicate with each other. It is implemented as a Python dictionary, available on the `job` instance passed to tasks as `job.workspace`. The initial workspace of a job can be empty, or can contain some parameters that the tasks require (for example, API access tokens, account IDs etc). A single task can edit the workspace, and the modified workspace will be passed on to the next task in the sequence. For example: @@ -32,7 +71,13 @@ The *workspace* is an area that tasks within a single job can use to communicate def my_second_task(job): logger.info("Task 1 says: %s" % job.workspace['message']) -#### Worker process +When creating a Job, the workspace is passed as a keyword argument: + +```python +Job.objects.create(name='my_job', workspace={'key': value}) +``` + +### Worker process A *worker process* is a long-running process, implemented as a Django management command, which is responsible for executing the tasks associated with a job. There may be many worker processes running concurrently in the final system. Worker processes wait for a new job to be created in the database, and call the each associated task in the correct sequeunce.. A worker can be started using `python manage.py worker`, and a single worker instance is included in the development `procfile`. From a0249c9ecbf0b868763dde3ac99c5a62403cf0ae Mon Sep 17 00:00:00 2001 From: Pete Wildsmith Date: Thu, 19 Mar 2015 15:34:15 +0000 Subject: [PATCH 2/4] replace hats with cats --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1f9bef9..d5410c4 100644 --- a/README.md +++ b/README.md @@ -83,10 +83,10 @@ A *worker process* is a long-running process, implemented as a Django management ### Configuration -Jobs are configured in the Django `settings.py` file. The `JOBS` setting is a dictionary mapping a *job name* (eg `import_hats`) to a *list* of one or more task function paths. For example: +Jobs are configured in the Django `settings.py` file. The `JOBS` setting is a dictionary mapping a *job name* (eg `import_cats`) to a *list* of one or more task function paths. For example: JOBS = { - 'import_hats': ['apps.hat_hatter.import_hats.step_one', 'apps.hat_hatter.import_hats.step_two'], + 'import_cats': ['apps.cat_importer.import_cats.step_one', 'apps.cat_importer.import_cats.step_two'], } ### Job states From 0ea8799db402035751fcbad9d8b2ec2cf0e54f6b Mon Sep 17 00:00:00 2001 From: Pete Wildsmith Date: Thu, 19 Mar 2015 15:35:58 +0000 Subject: [PATCH 3/4] include current PyPI release badge in README --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index d5410c4..afc5419 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,7 @@ django-db-queue ========== [![Build Status](https://travis-ci.org/dabapps/django-db-queue.svg)](https://travis-ci.org/dabapps/django-db-queue) +![pypi release](https://img.shields.io/pypi/v/django-db-queue.svg) Simple databased-backed job queue. Jobs are defined in your settings, and are processed by management commands. From b51ff85031223d3138665ddf096129c1a8383211 Mon Sep 17 00:00:00 2001 From: Pete Wildsmith Date: Thu, 19 Mar 2015 15:37:49 +0000 Subject: [PATCH 4/4] a PyPI release badge that links to PyPI --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index afc5419..8deada1 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ django-db-queue ========== [![Build Status](https://travis-ci.org/dabapps/django-db-queue.svg)](https://travis-ci.org/dabapps/django-db-queue) -![pypi release](https://img.shields.io/pypi/v/django-db-queue.svg) +[![pypi release](https://img.shields.io/pypi/v/django-db-queue.svg)](https://pypi.python.org/pypi/django-db-queue) Simple databased-backed job queue. Jobs are defined in your settings, and are processed by management commands.