This repository has been archived by the owner on Jul 22, 2019. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Adds support for creating Celery applications with Flask-AppFactory. * Increases test coverage to 100%. Signed-off-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
- Loading branch information
Showing
18 changed files
with
236 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,89 @@ | ||
Step 4: Celery based background tasks | ||
===================================== | ||
|
||
Flask-AppFactory includes optional support for Celery integration via the | ||
`Flask-CeleryExt <http://flask-celeryext.readthedocs.org>`_ extension. If you | ||
wish to use it, be sure to install Flask-AppFactory like this:: | ||
|
||
pip install Flask-AppFactory[celery] | ||
|
||
To enable Celery support we add one file to our application package, and one | ||
file to our reusable package:: | ||
|
||
myapp/celery.py | ||
mymodule/tasks.py | ||
|
||
|
||
Configuration | ||
------------- | ||
In our configuration we first have to add the Flask-CeleryExt extension | ||
(line 5) as well as define the Celery broker via the ``BROKER_URL`` variable | ||
(line 13). Note in this example we use a local Redis installation. For other | ||
options see the Celery documentation. | ||
|
||
.. literalinclude:: ../examples/myapp/config.py | ||
:language: python | ||
:linenos: | ||
:emphasize-lines: 5, 13 | ||
|
||
The Flask-CeleryExt takes care of creating a minimal Celery application with | ||
the correct configuration so Celery knows e.g. which broker to use. In addition | ||
the minimal Celery application doesn't load any tasks to ensure faster startup | ||
time. | ||
|
||
See the `Celery documentation | ||
<http://celery.readthedocs.org/en/latest/configuration.html>`_ for all the | ||
possible configuration variables. | ||
|
||
Celery application | ||
------------------ | ||
Next, we create a small Celery application which is later used to start the | ||
Celery worker from. | ||
|
||
.. literalinclude:: ../examples/myapp/celery.py | ||
:language: python | ||
:linenos: | ||
|
||
Reusable tasks | ||
-------------- | ||
The reusable packages can easily define Celery tasks by adding a ``tasks.py`` | ||
file like this: | ||
|
||
.. literalinclude:: ../examples/mymodule/tasks.py | ||
:language: python | ||
:linenos: | ||
:emphasize-lines: 8, 13 | ||
|
||
|
||
Notice the use of ``@shared_task`` decorator (line 8). This ensures that the | ||
task can be reused by many different Celery applications. The Celery | ||
application created above takes care of register the tasks. | ||
|
||
Each task is executed within a Flask application context (notice the use of | ||
e.g. ``current_app``). If you need to have a task executed in a request context | ||
(e.g. if you need to ensure before first request functions have been run), you | ||
just have to change the base class to use the ``RequestContextTask`` (line 13). | ||
|
||
Running the worker | ||
------------------ | ||
|
||
Next, you can start a Celery worker by simply pointing Celery to your new | ||
Celery application in ``myapp.celery``: | ||
|
||
.. code-block:: console | ||
$ celery worker -A myapp.celery | ||
Sending tasks | ||
------------- | ||
|
||
Executing tasks requires you to have your Flask application initialized, hence | ||
simply start a Python shell using your management script: | ||
|
||
.. code-block:: console | ||
$ myapp shell | ||
... | ||
>>> from mymodule.tasks import appctx | ||
>>> appctx.delay() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# myapp/celery.py | ||
|
||
from flask_appfactory.celery import celeryfactory | ||
from .app import create_app | ||
|
||
celery = celeryfactory(create_app()) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# mymodule/tasks.py | ||
|
||
from celery import shared_task | ||
from flask import current_app, request | ||
from flask_celeryext import RequestContextTask | ||
|
||
|
||
@shared_task | ||
def appctx(): | ||
current_app.logger.info(current_app.config['BROKER_URL']) | ||
|
||
|
||
@shared_task(base=RequestContextTask) | ||
def reqctx(): | ||
current_app.logger.info(request.method) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -15,7 +15,7 @@ | |
}, | ||
install_requires=[ | ||
'Flask>=0.10', | ||
'Flask-AppFactory', | ||
'Flask-AppFactory[celery]', | ||
'Flask-SQLAlchemy', | ||
], | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
# -*- coding: utf-8 -*- | ||
# | ||
# This file is part of Flask-AppFactory | ||
# Copyright (C) 2015 CERN. | ||
# | ||
# Flask-AppFactory is free software; you can redistribute it and/or | ||
# modify it under the terms of the Revised BSD License; see LICENSE | ||
# file for more details. | ||
|
||
"""Celery application factory.""" | ||
|
||
from __future__ import absolute_import, print_function, unicode_literals | ||
|
||
from flask_celeryext import create_celery_app | ||
from flask_registry import ModuleAutoDiscoveryRegistry | ||
|
||
|
||
def load_tasks(app): | ||
"""Load Celery tasks from installed packages.""" | ||
app.extensions['registry']['tasks'] = ModuleAutoDiscoveryRegistry( | ||
module_name='tasks', app=app) | ||
|
||
|
||
def celeryfactory(app): | ||
"""Create a Celery application based on Flask application. | ||
:param app: Flask application instance. | ||
""" | ||
try: | ||
# Check if celery application has already been created. | ||
celery = app.extensions['flask-celeryext'].celery | ||
except KeyError: | ||
celery = create_celery_app(app) | ||
load_tasks(app) | ||
|
||
return celery |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,4 @@ | ||
-e git+https://github.com/mitsuhiko/flask.git#egg=Flask | ||
-e git+https://github.com/inveniosoftware/flask-registry.git#egg=Flask-Registry | ||
-e git+https://github.com/inveniosoftware/flask-cli.git#egg=Flask-CLI | ||
-e git+https://github.com/inveniosoftware/flask-celeryext.git#egg=Flask-CeleryExt |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,3 +2,4 @@ Flask | |
click | ||
Flask-Registry | ||
Flask-CLI | ||
Flask-CeleryExt |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,3 +2,4 @@ Flask==0.10 | |
click==2.0 | ||
Flask-Registry==0.2.0 | ||
Flask-CLI==0.2.1 | ||
Flask-CeleryExt==0.1.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
# -*- coding: utf-8 -*- | ||
# | ||
# This file is part of Flask-AppFactory | ||
# Copyright (C) 2015 CERN. | ||
# | ||
# Flask-AppFactory is free software; you can redistribute it and/or | ||
# modify it under the terms of the Revised BSD License; see LICENSE | ||
# file for more details. | ||
|
||
from celery import shared_task | ||
|
||
|
||
@shared_task | ||
def simpletask(): | ||
"""Simple test task.""" | ||
return "simpletask" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
# -*- coding: utf-8 -*- | ||
# | ||
# This file is part of Flask-AppFactory | ||
# Copyright (C) 2015 CERN. | ||
# | ||
# Flask-AppFactory is free software; you can redistribute it and/or | ||
# modify it under the terms of the Revised BSD License; see LICENSE | ||
# file for more details. | ||
|
||
"""Test celery factory.""" | ||
|
||
from __future__ import absolute_import, print_function, unicode_literals | ||
|
||
from flask_appfactory import appfactory | ||
from flask_appfactory.celery import celeryfactory | ||
|
||
|
||
def test_dummy_app(): | ||
""""Test celery app creation.""" | ||
class conf: | ||
EXTENSIONS = ['flask_celeryext:FlaskCeleryExt'] | ||
|
||
app = appfactory("app2", conf) | ||
celery = celeryfactory(app) | ||
assert celery | ||
assert celery.flask_app == app | ||
assert app.extensions['flask-celeryext'].celery == celery | ||
|
||
|
||
def test_dummy_app_noext(): | ||
""""Test celery app creation without extension.""" | ||
app = appfactory("app3", None) | ||
celery = celeryfactory(app) | ||
assert celery | ||
assert celery.flask_app == app | ||
assert 'flask-celeryext' not in app.extensions |