Browse files

Merge branch '3.0'

  • Loading branch information...
ask committed Nov 5, 2012
2 parents 10fe588 + 3ff75c6 commit 5cb59d018e8992689afdd8cf1c97234f0c1c2cb7
Showing with 70 additions and 230 deletions.
  1. +29 −2 Changelog
  2. +37 −224 docs/userguide/monitoring.rst
  3. +2 −2 requirements/default.txt
  4. +2 −2 setup.cfg
@@ -26,10 +26,35 @@ If you're looking for versions prior to 3.0.x you should go to :ref:`history`.
:release-date: TBA
+- Now depends on kombu 2.4.8
+ - [Redis] New and improved fair queue cycle algorithm (Kevin McCarthy).
+ - [Redis] Now uses a Redis-based mutex when restoring messages.
+ - [Redis] Number of messages that can be restored in one interval is no
+ longer limited (but can be set using the
+ ``unacked_restore_limit`` transport option.)
+ - Heartbeat value can be specified in broker URLs (Mher Movsisyan).
+ - Fixed problem with msgpack on Python 3 (Jasper Bryant-Greene).
+- Now depends on billiard
+- Celery can now be used with static analysis tools like PyDev/PyCharm/pylint
+ etc.
+- Development documentation has moved to Read The Docs.
+ The new URL is:
- Event state's ``tasks_by_name`` applied limit before filtering by name.
Fix contributed by Alexander A. Sosnovskiy.
+- New :setting:`CELERY_QUEUE_HA_POLICY` setting used to set the default
+ HA policy for queues when using RabbitMQ.
+- New method ``Task.subtask_from_request`` returns a subtask using the current
+ request.
- Results get_many method did not respect timeout argument.
Fix contributed by Remigiusz Modrzejewski
@@ -57,6 +82,8 @@ If you're looking for versions prior to 3.0.x you should go to :ref:`history`.
- :mod:`celery.contrib.batches` now works again.
>>>>>>> 3.0
+- Fixed missing whitespace in ``bdist_rpm`` requirements (Issue #1046).
.. _version-3.0.11:
@@ -116,10 +143,10 @@ If you're looking for versions prior to 3.0.x you should go to :ref:`history`.
- Terminating a task now properly updates the state of the task to revoked,
and sends a ``task-revoked`` event.
-- Multi: No longer parses --app option (Issue #1008).
- Generic worker init script now waits for workers to shutdown by default.
+- Multi: No longer parses --app option (Issue #1008).
- Multi: stop_verify command renamed to stopwait.
- Daemonization: Now delays trying to create pidfile/logfile until after
@@ -171,30 +171,40 @@ You can specify a single, or a list of workers by using the
.. _monitoring-flower:
-Celery Flower: Web interface
+Flower: Real-time Celery web-monitor
-Celery Flower is a web based, real-time monitor and administration tool.
+Flower is a real-time web based monitor and administration tool for Celery.
+It is under active development, but is already an essential tool.
+Being the recommended monitor for Celery, it obsoletes the Django-Admin
+monitor, celerymon and the ncurses based monitor.
+Flower is pronounced like "flow", but you can also use the botanical version
+if you prefer.
-- Shutdown or restart workers
-- View workers status (completed, running tasks, etc.)
-- View worker pool options (timeouts, processes, etc.)
-- Control worker pool size
-- View message broker options
-- View active queues, add or cancel queues
-- View processed task stats by type
-- View currently running tasks
-- View scheduled tasks
-- View reserved and revoked tasks
-- Apply time and rate limits
-- View all active configuration options
-- View all tasks (by type, by worker, etc.)
-- View all task options (arguments, start time, runtime, etc.)
-- Revoke or terminate tasks
-- View real-time execution graphs
+- Real-time monitoring using Celery Events
+ - Task progress and history.
+ - Ability to show task details (arguments, start time, runtime, and more)
+ - Graphs and statistics
+- Remote Control
+ - View worker status and statistics.
+ - Shutdown and restart worker instances.
+ - Control worker pool size and autoscale settings.
+ - View and modify the queues a worker instance consumes from.
+ - View currently running tasks
+ - View scheduled tasks (ETA/countdown)
+ - View reserved and revoked tasks
+ - Apply time and rate limits
+ - Configuration viewer
+ - Revoke or terminate tasks
@@ -211,203 +221,22 @@ More screenshots_:
-Install Celery Flower:
+You can use pip to install Flower:
.. code-block:: bash
$ pip install flower
-Launch Celery Flower and open http://localhost:5555 in browser:
+Running the flower command will start a web-server that you can visit:
.. code-block:: bash
$ celery flower
-.. _monitoring-django-admin:
-Django Admin Monitor
-.. versionadded:: 2.1
-When you add `django-celery`_ to your Django project you will
-automatically get a monitor section as part of the Django admin interface.
-This can also be used if you're not using Celery with a Django project.
-.. figure:: ../images/djangoceleryadmin2.jpg
- :width: 700px
-.. _`django-celery`:
-.. _monitoring-django-starting:
-Starting the monitor
-The Celery section will already be present in your admin interface,
-but you won't see any data appearing until you start the snapshot camera.
-The camera takes snapshots of the events your workers sends at regular
-intervals, storing them in your database (See :ref:`monitoring-snapshots`).
-To start the camera run:
-.. code-block:: bash
- $ python celerycam
-If you haven't already enabled the sending of events you need to do so:
-.. code-block:: bash
- $ python celery control enable_events
-:Tip: You can enable events when the worker starts using the `-E` argument.
-Now that the camera has been started, and events have been enabled
-you should be able to see your workers and the tasks in the admin interface
-(it may take some time for workers to show up).
-The admin interface shows tasks, worker nodes, and even
-lets you perform some actions, like revoking and rate limiting tasks,
-or shutting down worker nodes.
-.. _monitoring-django-frequency:
-Shutter frequency
-By default the camera takes a snapshot every second, if this is too frequent
-or you want to have higher precision, then you can change this using the
-``--frequency`` argument. This is a float describing how often, in seconds,
-it should wake up to check if there are any new events:
-.. code-block:: bash
- $ python celerycam --frequency=3.0
-The camera also supports rate limiting using the ``--maxrate`` argument.
-While the frequency controls how often the camera thread wakes up,
-the rate limit controls how often it will actually take a snapshot.
-The rate limits can be specified in seconds, minutes or hours
-by appending `/s`, `/m` or `/h` to the value.
-Example: ``--maxrate=100/m``, means "hundred writes a minute".
-The rate limit is off by default, which means it will take a snapshot
-for every ``--frequency`` seconds.
-The events also expire after some time, so the database doesn't fill up.
-Successful tasks are deleted after 1 day, failed tasks after 3 days,
-and tasks in other states after 5 days.
-.. _monitoring-django-reset:
-Resetting monitor data
+The default port is http://localhost:5555, but you can change this using the
+:option:`--port` argument::
-To reset the monitor data you need to clear out two models::
- >>> from djcelery.models import WorkerState, TaskState
- # delete worker history
- >>> WorkerState.objects.all().delete()
- # delete task history
- >>> TaskState.objects.all().update(hidden=True)
- >>> TaskState.objects.purge()
-.. _monitoring-django-expiration:
-By default monitor data for successful tasks will expire in 1 day,
-failed tasks in 3 days and pending tasks in 5 days.
-You can change the expiry times for each of these using
-adding the following settings to your :file:``:
-.. code-block:: python
- from datetime import timedelta
- CELERYCAM_EXPIRE_SUCCESS = timedelta(hours=1)
- CELERYCAM_EXPIRE_ERROR = timedelta(hours=2)
- CELERYCAM_EXPIRE_PENDING = timedelta(hours=2)
-.. _monitoring-nodjango:
-Using outside of Django
-`django-celery` also installs the :program:`djcelerymon` program. This
-can be used by non-Django users, and runs both a web server and a snapshot
-camera in the same process.
-Using :program:`pip`:
-.. code-block:: bash
- $ pip install -U django-celery
-or using :program:`easy_install`:
-.. code-block:: bash
- $ easy_install -U django-celery
-:program:`djcelerymon` reads configuration from your Celery configuration
-module, and sets up the Django environment using the same settings:
-.. code-block:: bash
- $ djcelerymon
-Database tables will be created the first time the monitor is run.
-By default an `sqlite3` database file named
-:file:`djcelerymon.db` is used, so make sure this file is writeable by the
-user running the monitor.
-If you want to store the events in a different database, e.g. MySQL,
-then you can configure the `DATABASE*` settings directly in your Celery
-config module. See
-for more information about the database options available.
-You will also be asked to create a superuser (and you need to create one
-to be able to log into the admin later)::
- Creating table auth_permission
- Creating table auth_group_permissions
- [...]
- You just installed Django's auth system, which means you don't
- have any superusers defined. Would you like to create
- one now? (yes/no): yes
- Username (Leave blank to use 'username'): username
- Email address:
- Password: ******
- Password (again): ******
- Superuser created successfully.
- [...]
- Django version 1.2.1, using settings 'celeryconfig'
- Development server is running at
- Quit the server with CONTROL-C.
-Now that the service is started you can visit the monitor
-at, and log in using the user you created.
-For a list of the command-line options supported by :program:`djcelerymon`,
-please see ``djcelerymon --help``.
+ $ open http://localhost:5555
.. _monitoring-celeryev:
@@ -419,7 +248,8 @@ celery events: Curses Monitor
`celery events` is a simple curses monitor displaying
task and worker history. You can inspect the result and traceback of tasks,
and it also supports some management commands like rate limiting and shutting
-down workers.
+down workers. This monitor was started as a proof of concept, and you
+probably want to use Flower instead.
@@ -451,23 +281,6 @@ For a complete list of options use ``--help``:
$ celery events --help
-.. _monitoring-celerymon:
-celerymon: Web monitor
-`celerymon`_ is the ongoing work to create a web monitor.
-It's far from complete yet, and does currently only support
-a JSON API. Help is desperately needed for this project, so if you,
-or someone you know would like to contribute templates, design, code
-or help this project in any way, please get in touch!
-:Tip: The Django admin monitor can be used even though you're not using
- Celery with a Django project. See :ref:`monitoring-nodjango`.
-.. _`celerymon`:
.. _monitoring-rabbitmq:
@@ -590,7 +403,7 @@ Events
The worker has the ability to send a message whenever some event
-happens. These events are then captured by tools like :program:`celerymon`
+happens. These events are then captured by tools like Flower,
and :program:`celery events` to monitor the cluster.
.. _monitoring-snapshots:
@@ -1,3 +1,3 @@
@@ -15,5 +15,5 @@ upload-dir = docs/.build/html
requires = pytz
- billiard >=
- kombu >= 2.4.7
+ billiard >=
+ kombu >= 2.4.8

0 comments on commit 5cb59d0

Please sign in to comment.