Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse code

Initial commit, based on django-gearman (http://github.com/fwenzel/dj…

  • Loading branch information...
commit 6aacd81c7e8c239ab4f8bf6924e79c0173ad7058 0 parents
Jonas authored June 12, 2010
3  CONTRIBUTORS
... ...
@@ -0,0 +1,3 @@
  1
+Jonas VP <jvp@jonasundderwolf.de>
  2
+Fred Wenzel <fwenzel@mozilla.com>
  3
+Jeff Balogh <me@jeffbalogh.org>
1  MANIFEST.in
... ...
@@ -0,0 +1 @@
  1
+include README.md
145  README.md
Source Rendered
... ...
@@ -0,0 +1,145 @@
  1
+django-beanstalkd
  2
+=================
  3
+
  4
+*django-beanstalkd* is a convenience wrapper for the [beanstalkd][beanstalkd]
  5
+[Python Bindings][beanstalkc].
  6
+
  7
+With django-beanstalkd, you can code jobs as well as clients in a Django project
  8
+with minimal overhead in your application. Server connections etc. all take
  9
+place in django-beanstalkd and don't unnecessarily clog your application code.
  10
+
  11
+This library is based in large part on Fred Wenzel's [django-gearman][django-gearman].
  12
+If you're looking for synchronous execution of jobs, check out [Gearman][gearman]
  13
+and Fred's library! Beanstalkd is useful for background processes only.
  14
+
  15
+[beanstalkd]: http://kr.github.com/beanstalkd/
  16
+[beanstalkc]: http://github.com/earl/beanstalkc/
  17
+[django-gearman]: http://github.com/fwenzel/django-gearman
  18
+[gearman]: http://gearman.org/
  19
+
  20
+Installation
  21
+------------
  22
+It's the same for both the client and worker instances of your django project:
  23
+
  24
+    pip install -e git://github.com/jonasvp/django-beanstalkd.git#egg=django-beanstalkd
  25
+
  26
+Add ``django_beanstalkd`` to the `INSTALLED_APPS` section of `settings.py`.
  27
+
  28
+Specify the following settings in your local settings.py file if your beanstalkd
  29
+server isn't accessible on port 11300 of localhost (127.0.0.1):
  30
+
  31
+    # My beanstalkd server
  32
+    BEANSTALK_SERVER = '127.0.0.1:11300'  # the default value
  33
+
  34
+If necessary, you can specify a pattern to be applied to your beanstalk worker
  35
+functions:
  36
+
  37
+    # beanstalk job name pattern. Namespacing etc goes here. This is the pattern
  38
+    # your jobs will register as with the server, and that you'll need to use
  39
+    # when calling them from a non-django-beanstalkd client.
  40
+    # replacement patterns are:
  41
+    # %(app)s : django app name the job is filed under
  42
+    # %(job)s : job name
  43
+    BEANSTALK_JOB_NAME = '%(app)s.%(job)s'
  44
+
  45
+
  46
+Workers
  47
+-------
  48
+### Registering jobs
  49
+Create a file `beanstalk_jobs.py` in any of your django apps, and define as many
  50
+jobs as functions as you like. The job must accept a single string argument as
  51
+passed by the caller.
  52
+
  53
+Mark each of these functions as beanstalk jobs by decorating them with
  54
+`django_beanstalkd.beanstalk_job`.
  55
+
  56
+For an example, look at the `beanstalk_example` app's `benstalk_jobs.py` file.
  57
+
  58
+### Starting a worker
  59
+To start a worker, run `python manage.py beanstalk_worker`. It will start
  60
+serving all registered jobs.
  61
+
  62
+To spawn more than one worker (if, e.g., most of your jobs are I/O bound),
  63
+use the `-w` option:
  64
+
  65
+    python manage.py beanstalk_worker -w 5
  66
+
  67
+will start five workers.
  68
+
  69
+Since the process will keep running while waiting for and executing jobs,
  70
+you probably want to run this in a _screen_ session or similar.
  71
+
  72
+Clients
  73
+-------
  74
+To make your workers work, you need a client app passing data to them. Create
  75
+and instance of the `django_beanstalkd.BeanstalkClient` class and `call` a
  76
+function with it:
  77
+
  78
+    from django_beanstalkd import BeanstalkClient
  79
+    client = BeanstalkClient()
  80
+    client.call('beanstalk_example.background_counting', '5')
  81
+
  82
+For a live example look at the `beanstalk_example` app, in the
  83
+`management/commands/beanstalk_example_client.py` file. Arguments to `call` are
  84
+
  85
+    priority: an integer number that specifies the priority. Jobs with a
  86
+              smaller priority get executed first
  87
+    delay:    how many seconds to wait before the job can be reserved
  88
+    ttr:      how many seconds a worker has to process the job before it gets requeued
  89
+
  90
+
  91
+Example App
  92
+-----------
  93
+For a full, working, example application, add `beanstalk_example` to your
  94
+`INSTALLED_APPS`, then run a worker in one shell:
  95
+
  96
+    python manage.py beanstalk_worker -w 4
  97
+
  98
+and execute the example app in another:
  99
+
  100
+    python manage.py beanstalk_example_client
  101
+
  102
+You can see the client sending data and the worker(s) working on it.
  103
+
  104
+Licensing
  105
+---------
  106
+This software is licensed under the [Mozilla Tri-License][MPL]:
  107
+
  108
+    ***** BEGIN LICENSE BLOCK *****
  109
+    Version: MPL 1.1/GPL 2.0/LGPL 2.1
  110
+
  111
+    The contents of this file are subject to the Mozilla Public License Version
  112
+    1.1 (the "License"); you may not use this file except in compliance with
  113
+    the License. You may obtain a copy of the License at
  114
+    http://www.mozilla.org/MPL/
  115
+
  116
+    Software distributed under the License is distributed on an "AS IS" basis,
  117
+    WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
  118
+    for the specific language governing rights and limitations under the
  119
+    License.
  120
+
  121
+    The Original Code is django-gearman.
  122
+
  123
+    The Initial Developer of the Original Code is Mozilla.
  124
+    Portions created by the Initial Developer are Copyright (C) 2010
  125
+    the Initial Developer. All Rights Reserved.
  126
+
  127
+    Contributor(s):
  128
+      Jonas VP <jvp@jonasundderwolf.de>
  129
+      Frederic Wenzel <fwenzel@mozilla.com>
  130
+
  131
+    Alternatively, the contents of this file may be used under the terms of
  132
+    either the GNU General Public License Version 2 or later (the "GPL"), or
  133
+    the GNU Lesser General Public License Version 2.1 or later (the "LGPL"),
  134
+    in which case the provisions of the GPL or the LGPL are applicable instead
  135
+    of those above. If you wish to allow use of your version of this file only
  136
+    under the terms of either the GPL or the LGPL, and not to allow others to
  137
+    use your version of this file under the terms of the MPL, indicate your
  138
+    decision by deleting the provisions above and replace them with the notice
  139
+    and other provisions required by the GPL or the LGPL. If you do not delete
  140
+    the provisions above, a recipient may use your version of this file under
  141
+    the terms of any one of the MPL, the GPL or the LGPL.
  142
+
  143
+    ***** END LICENSE BLOCK *****
  144
+
  145
+[MPL]: http://www.mozilla.org/MPL/
0  beanstalk_example/__init__.py
No changes.
21  beanstalk_example/beanstalk_jobs.py
... ...
@@ -0,0 +1,21 @@
  1
+"""
  2
+Example Beanstalk Job File.
  3
+Needs to be called beanstalk_jobs.py and reside inside a registered Django app.
  4
+"""
  5
+import os
  6
+import time
  7
+
  8
+from django_beanstalkd import beanstalk_job
  9
+
  10
+
  11
+@beanstalk_job
  12
+def background_counting(arg):
  13
+    """
  14
+    Do some incredibly useful counting to the value of arg
  15
+    """
  16
+    value = int(arg)
  17
+    pid = os.getpid()
  18
+    print "[%s] Counting from 1 to %d." % (pid, value)
  19
+    for i in range(1, value+1):
  20
+        print '[%s] %d' % (pid, i)
  21
+        time.sleep(1)
0  beanstalk_example/management/__init__.py
No changes.
0  beanstalk_example/management/commands/__init__.py
No changes.
18  beanstalk_example/management/commands/beanstalk_example_client.py
... ...
@@ -0,0 +1,18 @@
  1
+from django.core.management.base import NoArgsCommand
  2
+from django_beanstalkd import BeanstalkClient
  3
+
  4
+
  5
+class Command(NoArgsCommand):
  6
+    help = "Execute an example command with the django_beanstalk_jobs interface"
  7
+    __doc__ = help
  8
+
  9
+    def handle_noargs(self, **options):
  10
+        client = BeanstalkClient()
  11
+
  12
+        print "Asynchronous Beanstalk Call"
  13
+        print "-------------------------"
  14
+        print "Notice how this app exits, while the workers still work on the tasks."
  15
+        for i in range(4):
  16
+            client.call(
  17
+                'beanstalk_example.background_counting', '5'
  18
+            )
46  django_beanstalkd/__init__.py
... ...
@@ -0,0 +1,46 @@
  1
+"""
  2
+Django Beanstalk Interface
  3
+"""
  4
+from django.conf import settings
  5
+
  6
+from beanstalkc import Connection, SocketError, DEFAULT_PRIORITY, DEFAULT_TTR
  7
+
  8
+from decorators import beanstalk_job
  9
+
  10
+def connect_beanstalkd():
  11
+    """Connect to beanstalkd server(s) from settings file"""
  12
+
  13
+    server = getattr(settings, 'BEANSTALK_SERVER', '127.0.0.1')
  14
+    port = 11300
  15
+    if server.find(':') > -1:
  16
+        server, port = server.split(':', 1)
  17
+        
  18
+    try:
  19
+        port = int(port)
  20
+        return Connection(server, port)
  21
+    except (ValueError, SocketError), e:
  22
+        raise BeanstalkError(e)
  23
+
  24
+
  25
+class BeanstalkError(Exception):
  26
+    pass
  27
+
  28
+
  29
+class BeanstalkClient(object):
  30
+    """beanstalk client, automatically connecting to server"""
  31
+
  32
+    def call(self, func, arg='', priority=DEFAULT_PRIORITY, delay=0, ttr=DEFAULT_TTR):
  33
+        """
  34
+        Calls the specified function (in beanstalk terms: put the specified arg
  35
+        in tube func)
  36
+        
  37
+        priority: an integer number that specifies the priority. Jobs with a
  38
+                  smaller priority get executed first
  39
+        delay: how many seconds to wait before the job can be reserved
  40
+        ttr: how many seconds a worker has to process the job before it gets requeued
  41
+        """
  42
+        self._beanstalk.use(func)
  43
+        self._beanstalk.put(str(arg), priority=priority, delay=delay, ttr=ttr)
  44
+
  45
+    def __init__(self, **kwargs):
  46
+        self._beanstalk = connect_beanstalkd()
27  django_beanstalkd/decorators.py
... ...
@@ -0,0 +1,27 @@
  1
+class beanstalk_job(object):
  2
+    """
  3
+    Decorator marking a function inside some_app/beanstalk_jobs.py as a
  4
+    beanstalk job
  5
+    """
  6
+
  7
+    def __init__(self, f):
  8
+        self.f = f
  9
+        self.__name__ = f.__name__
  10
+        
  11
+        # determine app name
  12
+        parts = f.__module__.split('.')
  13
+        if len(parts) > 1:
  14
+            self.app = parts[-2]
  15
+        else:
  16
+            self.app = ''
  17
+
  18
+        # store function in per-app job list (to be picked up by a worker)
  19
+        bs_module = __import__(f.__module__)
  20
+        try:
  21
+            bs_module.beanstalk_job_list.append(self)
  22
+        except AttributeError:
  23
+            bs_module.beanstalk_job_list = [self]
  24
+
  25
+    def __call__(self, arg):
  26
+        # call function with argument passed by the client only
  27
+        return self.f(arg)
0  django_beanstalkd/management/__init__.py
No changes.
0  django_beanstalkd/management/commands/__init__.py
No changes.
114  django_beanstalkd/management/commands/beanstalk_worker.py
... ...
@@ -0,0 +1,114 @@
  1
+from optparse import make_option
  2
+import os
  3
+import sys
  4
+
  5
+from django.conf import settings
  6
+from django.core.management.base import NoArgsCommand
  7
+from django_beanstalkd import connect_beanstalkd
  8
+
  9
+
  10
+class Command(NoArgsCommand):
  11
+    help = "Start a Beanstalk worker serving all registered Beanstalk jobs"
  12
+    __doc__ = help
  13
+    option_list = NoArgsCommand.option_list + (
  14
+        make_option('-w', '--workers', action='store', dest='worker_count',
  15
+                    default='1', help='Number of workers to spawn.'),
  16
+    )
  17
+    children = [] # list of worker processes
  18
+    jobs = {}
  19
+
  20
+    def handle_noargs(self, **options):
  21
+        # find beanstalk job modules
  22
+        bs_modules = []
  23
+        for app in settings.INSTALLED_APPS:
  24
+            try:
  25
+                bs_modules.append(__import__("%s.beanstalk_jobs" % app))
  26
+            except ImportError:
  27
+                pass
  28
+        if not bs_modules:
  29
+            print "No beanstalk_jobs modules found!"
  30
+            return
  31
+
  32
+        # find all jobs
  33
+        jobs = []
  34
+        for bs_module in bs_modules:
  35
+            try:
  36
+                bs_module.beanstalk_job_list
  37
+            except AttributeError:
  38
+                continue
  39
+            jobs += bs_module.beanstalk_job_list
  40
+        if not jobs:
  41
+            print "No beanstalk jobs found!"
  42
+            return
  43
+        print "Available jobs:"
  44
+        for job in jobs:
  45
+            # determine right name to register function with
  46
+            app = job.app
  47
+            jobname = job.__name__
  48
+            try:
  49
+                func = settings.BEANSTALK_JOB_NAME % {
  50
+                    'app': app,
  51
+                    'job': jobname,
  52
+                }
  53
+            except AttributeError:
  54
+                func = '%s.%s' % (app, jobname)
  55
+            self.jobs[func] = job
  56
+            print "* %s" % func
  57
+
  58
+        # spawn all workers and register all jobs
  59
+        try:
  60
+            worker_count = int(options['worker_count'])
  61
+            assert(worker_count > 0)
  62
+        except (ValueError, AssertionError):
  63
+            worker_count = 1
  64
+        self.spawn_workers(worker_count)
  65
+
  66
+        # start working
  67
+        print "Starting to work... (press ^C to exit)"
  68
+        try:
  69
+            for child in self.children:
  70
+                os.waitpid(child, 0)
  71
+        except KeyboardInterrupt:
  72
+            sys.exit(0)
  73
+
  74
+    def spawn_workers(self, worker_count):
  75
+        """
  76
+        Spawn as many workers as desired (at least 1).
  77
+        Accepts:
  78
+        - worker_count, positive int
  79
+        """
  80
+        # no need for forking if there's only one worker
  81
+        if worker_count == 1:
  82
+            return self.work()
  83
+
  84
+        print "Spawning %s worker(s)" % worker_count
  85
+        # spawn children and make them work (hello, 19th century!)
  86
+        for i in range(worker_count):
  87
+            child = os.fork()
  88
+            if child:
  89
+                self.children.append(child)
  90
+                continue
  91
+            else:
  92
+                self.work()
  93
+                break
  94
+
  95
+    def work(self):
  96
+        """children only: watch tubes for all jobs, start working"""
  97
+        beanstalk = connect_beanstalkd()
  98
+        for job in self.jobs.keys():
  99
+            beanstalk.watch(job)
  100
+        beanstalk.ignore('default')
  101
+        
  102
+        try:
  103
+            while True:
  104
+                job = beanstalk.reserve()
  105
+                job_name = job.stats()['tube']
  106
+                if job_name in self.jobs:
  107
+                    #print "Calling %s with arg: %s" % (job_name, job.body)
  108
+                    self.jobs[job_name](job.body)
  109
+                    job.delete()
  110
+                else:
  111
+                    job.release()
  112
+            
  113
+        except KeyboardInterrupt:
  114
+            sys.exit(0)
4  requirements.txt
... ...
@@ -0,0 +1,4 @@
  1
+# pip requirements file
  2
+pyyaml
  3
+beanstalkc
  4
+
27  setup.py
... ...
@@ -0,0 +1,27 @@
  1
+from setuptools import setup, find_packages
  2
+
  3
+
  4
+setup(
  5
+    name='django-beanstalkd',
  6
+    version='0.1',
  7
+    description='A convenience wrapper for beanstalkd clients and workers '
  8
+                'in Django using the beanstalkc library for Python',
  9
+    long_description=open('README.md').read(),
  10
+    author='Jonas VP',
  11
+    author_email='jvp@jonasundderwolf.de',
  12
+    url='http://github.com/jonasvp/django-beanstalkd',
  13
+    license='MPL',
  14
+    packages=find_packages(),
  15
+    include_package_data=True,
  16
+    zip_safe=False,
  17
+    install_requires=['pyyaml', 'beanstalkc'],
  18
+    classifiers=[
  19
+        'Development Status :: 4 - Beta',
  20
+        'Environment :: Web Environment',
  21
+        'Framework :: Django',
  22
+        'Intended Audience :: Developers',
  23
+        'Operating System :: OS Independent',
  24
+        'Programming Language :: Python',
  25
+        'Topic :: Software Development :: Libraries :: Python Modules',
  26
+    ]
  27
+)

0 notes on commit 6aacd81

Please sign in to comment.
Something went wrong with that request. Please try again.