celery: Flask integration

* Adds support for creating Celery applications with Flask-AppFactory.

* Increases test coverage to 100%.

Signed-off-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>

docs/api.rst

@@ -16,6 +16,12 @@ CLI factory
    :members:
    :undoc-members:
 
+Celery factory
+--------------
+.. automodule:: flask_appfactory.celery
+   :members:
+   :undoc-members:
+
 Extensions
 ----------
 

docs/tut_app.rst

@@ -31,7 +31,9 @@ translations etc. is loaded according to this order.
 The ``EXTENSIONS`` defines the list of extensions that Flask-AppFactory
 should load. You'll see that in addition to ``myexts.sqlalchemy`` we also
 load ``flask_appfactory.ext.jinja2``. This extension is needed in order to make
-the template loader aware of the order of packages.
+the template loader aware of the order of packages. We also load
+``flask_celeryext:FlaskCeleryExt``, which will be explained in detail in
+Step 4 together with ``BROKER_URL``.
 
 
 Application factory

docs/tut_celery.rst

@@ -0,0 +1,89 @@
+Step 4: Celery based background tasks
+=====================================
+
+Flask-AppFactory includes optional support for Celery integration via the
+`Flask-CeleryExt <http://flask-celeryext.readthedocs.org>`_ extension. If you
+wish to use it, be sure to install Flask-AppFactory like this::
+
+    pip install Flask-AppFactory[celery]
+
+To enable Celery support we add one file to our application package, and one
+file to our reusable package::
+
+    myapp/celery.py
+    mymodule/tasks.py
+
+
+Configuration
+-------------
+In our configuration we first have to add the Flask-CeleryExt extension
+(line 5) as well as define the Celery broker via the ``BROKER_URL`` variable
+(line 13). Note in this example we use a local Redis installation. For other
+options see the Celery documentation.
+
+.. literalinclude:: ../examples/myapp/config.py
+   :language: python
+   :linenos:
+   :emphasize-lines: 5, 13
+
+The Flask-CeleryExt takes care of creating a minimal Celery application with
+the correct configuration so Celery knows e.g. which broker to use. In addition
+the minimal Celery application doesn't load any tasks to ensure faster startup
+time.
+
+See the `Celery documentation
+<http://celery.readthedocs.org/en/latest/configuration.html>`_ for all the
+possible configuration variables.
+
+Celery application
+------------------
+Next, we create a small Celery application which is later used to start the
+Celery worker from.
+
+.. literalinclude:: ../examples/myapp/celery.py
+   :language: python
+   :linenos:
+
+Reusable tasks
+--------------
+The reusable packages can easily define Celery tasks by adding a ``tasks.py``
+file like this:
+
+.. literalinclude:: ../examples/mymodule/tasks.py
+   :language: python
+   :linenos:
+   :emphasize-lines: 8, 13
+
+
+Notice the use of ``@shared_task`` decorator (line 8). This ensures that the
+task can be reused by many different Celery applications. The Celery
+application created above takes care of register the tasks.
+
+Each task is executed within a Flask application context (notice the use of
+e.g. ``current_app``). If you need to have a task executed in a request context
+(e.g. if you need to ensure before first request functions have been run), you
+just have to change the base class to use the ``RequestContextTask`` (line 13).
+
+Running the worker
+------------------
+
+Next, you can start a Celery worker by simply pointing Celery to your new
+Celery application in ``myapp.celery``:
+
+.. code-block:: console
+
+   $ celery worker -A myapp.celery
+
+
+Sending tasks
+-------------
+
+Executing tasks requires you to have your Flask application initialized, hence
+simply start a Python shell using your management script:
+
+.. code-block:: console
+
+   $ myapp shell
+   ...
+   >>> from mymodule.tasks import appctx
+   >>> appctx.delay()

docs/tutorial.rst

@@ -14,3 +14,4 @@ The full source code by viewed in the ``examples`` folder in the package or on
    tut_package
    tut_ext
    tut_app
+   tut_celery

examples/myapp/celery.py

@@ -0,0 +1,6 @@
+# myapp/celery.py
+
+from flask_appfactory.celery import celeryfactory
+from .app import create_app
+
+celery = celeryfactory(create_app())

examples/myapp/config.py

@@ -2,9 +2,12 @@
 
 EXTENSIONS = [
     "flask_appfactory.ext.jinja2",
+    "flask_celeryext:FlaskCeleryExt",
     "myexts.sqlalchemy",
 ]
 
 PACKAGES = [
     "mymodule",
 ]
+
+BROKER_URL = "redis://localhost:6379/0"

examples/mymodule/tasks.py

@@ -0,0 +1,15 @@
+# mymodule/tasks.py
+
+from celery import shared_task
+from flask import current_app, request
+from flask_celeryext import RequestContextTask
+
+
+@shared_task
+def appctx():
+    current_app.logger.info(current_app.config['BROKER_URL'])
+
+
+@shared_task(base=RequestContextTask)
+def reqctx():
+    current_app.logger.info(request.method)

examples/setup.py

@@ -15,7 +15,7 @@
     },
     install_requires=[
         'Flask>=0.10',
-        'Flask-AppFactory',
+        'Flask-AppFactory[celery]',
         'Flask-SQLAlchemy',
     ],
 )

flask_appfactory/app.py

@@ -162,7 +162,7 @@ def base_app(app_name, instance_path=None, static_folder=None,
     try:
         if not os.path.exists(instance_path):
             os.makedirs(instance_path)
-    except Exception:
+    except Exception:  # pragma: no cover
         pass
 
     # Create the Flask application instance

flask_appfactory/celery.py

@@ -0,0 +1,36 @@
+# -*- coding: utf-8 -*-
+#
+# This file is part of Flask-AppFactory
+# Copyright (C) 2015 CERN.
+#
+# Flask-AppFactory is free software; you can redistribute it and/or
+# modify it under the terms of the Revised BSD License; see LICENSE
+# file for more details.
+
+"""Celery application factory."""
+
+from __future__ import absolute_import, print_function, unicode_literals
+
+from flask_celeryext import create_celery_app
+from flask_registry import ModuleAutoDiscoveryRegistry
+
+
+def load_tasks(app):
+    """Load Celery tasks from installed packages."""
+    app.extensions['registry']['tasks'] = ModuleAutoDiscoveryRegistry(
+            module_name='tasks', app=app)
+
+
+def celeryfactory(app):
+    """Create a Celery application based on Flask application.
+
+    :param app: Flask application instance.
+    """
+    try:
+        # Check if celery application has already been created.
+        celery = app.extensions['flask-celeryext'].celery
+    except KeyError:
+        celery = create_celery_app(app)
+    load_tasks(app)
+
+    return celery

flask_appfactory/ext/jinja2.py

@@ -36,7 +36,7 @@
 try:
     # Deprecated in Flask commit 817b72d484d353800d907b3580c899314bf7f3c6
     from flask.templating import blueprint_is_module
-except ImportError:
+except ImportError:  # pragma: no cover
     def blueprint_is_module(blueprint):
         """Dummy function for Flask 1.0."""
         return False
@@ -54,11 +54,11 @@ class OrderAwareDispatchingJinjaLoader(DispatchingJinjaLoader):
     def _iter_loaders(self, template):
         for blueprint in self.app.extensions['registry']['blueprints']:
             if blueprint_is_module(blueprint):
-                continue
+                continue  # pragma: no cover
             loader = blueprint.jinja_loader
             if loader is not None:
                 if IS_FLASK_1_0:
-                    yield blueprint, loader
+                    yield blueprint, loader  # pragma: no cover
                 else:
                     yield loader, template
 

requirements.devel.txt

@@ -1,3 +1,4 @@
 -e git+https://github.com/mitsuhiko/flask.git#egg=Flask
 -e git+https://github.com/inveniosoftware/flask-registry.git#egg=Flask-Registry
 -e git+https://github.com/inveniosoftware/flask-cli.git#egg=Flask-CLI
+-e git+https://github.com/inveniosoftware/flask-celeryext.git#egg=Flask-CeleryExt

requirements.latest.txt

@@ -2,3 +2,4 @@ Flask
 click
 Flask-Registry
 Flask-CLI
+Flask-CeleryExt

requirements.lowest.txt

@@ -2,3 +2,4 @@ Flask==0.10
 click==2.0
 Flask-Registry==0.2.0
 Flask-CLI==0.2.1
+Flask-CeleryExt==0.1.0

setup.py

@@ -84,7 +84,7 @@ def run_tests(self):
         'Flask-CLI>=0.2.1',
     ],
     extras_require={
-        'docs': ['sphinx_rtd_theme'],
+        'celery': ['Flask-CeleryExt>=0.1.0'],
     },
     cmdclass={'test': PyTest},
     classifiers=[

tests/simplemodule/tasks.py

@@ -0,0 +1,16 @@
+# -*- coding: utf-8 -*-
+#
+# This file is part of Flask-AppFactory
+# Copyright (C) 2015 CERN.
+#
+# Flask-AppFactory is free software; you can redistribute it and/or
+# modify it under the terms of the Revised BSD License; see LICENSE
+# file for more details.
+
+from celery import shared_task
+
+
+@shared_task
+def simpletask():
+    """Simple test task."""
+    return "simpletask"

tests/test_app.py

@@ -12,7 +12,11 @@
 from __future__ import absolute_import, print_function, unicode_literals
 
 import os
+import shutil
+import tempfile
+
 from flask_appfactory import appfactory
+from flask_appfactory.app import base_app
 
 
 def test_dummy_app():
@@ -95,6 +99,18 @@ class conf:
         assert rv.data == 'TEST'.encode('utf-8')
 
 
+def test_instance_path_creation():
+    """Test instance path creation."""
+    path = tempfile.mkdtemp()
+    try:
+        instance_path = os.path.join(path, "myinstance")
+        assert not os.path.exists(instance_path)
+        base_app('instance', instance_path=instance_path)
+        assert os.path.exists(instance_path)
+    finally:
+        shutil.rmtree(path)
+
+
 def test_simple_app_noload():
     """."""
     class conf:

tests/test_celery.py

@@ -0,0 +1,36 @@
+# -*- coding: utf-8 -*-
+#
+# This file is part of Flask-AppFactory
+# Copyright (C) 2015 CERN.
+#
+# Flask-AppFactory is free software; you can redistribute it and/or
+# modify it under the terms of the Revised BSD License; see LICENSE
+# file for more details.
+
+"""Test celery factory."""
+
+from __future__ import absolute_import, print_function, unicode_literals
+
+from flask_appfactory import appfactory
+from flask_appfactory.celery import celeryfactory
+
+
+def test_dummy_app():
+    """"Test celery app creation."""
+    class conf:
+        EXTENSIONS = ['flask_celeryext:FlaskCeleryExt']
+
+    app = appfactory("app2", conf)
+    celery = celeryfactory(app)
+    assert celery
+    assert celery.flask_app == app
+    assert app.extensions['flask-celeryext'].celery == celery
+
+
+def test_dummy_app_noext():
+    """"Test celery app creation without extension."""
+    app = appfactory("app3", None)
+    celery = celeryfactory(app)
+    assert celery
+    assert celery.flask_app == app
+    assert 'flask-celeryext' not in app.extensions