Uploads results of Celery tasks to AWS CloudWatch.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
celery_cloudwatch
.gitignore
.pylintrc
.scrutinizer.yml
LICENSE.md
MANIFEST.in
README.rst
requirements.txt
setup.cfg
setup.py

README.rst

https://scrutinizer-ci.com/g/SectorLabs/celery-cloudwatch/badges/build.png?b=master&s=dbee8972701eec8cb3f256360b6c8f2880a8eae7 https://scrutinizer-ci.com/g/SectorLabs/celery-cloudwatch/badges/quality-score.png?b=master&s=b715993dcceb3ec81785e324f3cb36b17c40fa0f

This is a simple monitoring tool that intercepts the results of Celery tasks and uploads the result to AWS CloudWatch.

Installation

Install from PyPi:

pip install celery-cloudwatch-logs

Configuration

Configuration is done through environment variables. The following environment variables must be set in order for celery-cloudwatch to work:

  • AWS_CLOUDWATCH_ACCESS_KEY
  • AWS_CLOUDWATCH_SECRET_KEY
  • AWS_CLOUDWATCH_GROUP_NAME

AWS_CLOUDWATCH_ACCESS_KEY and AWS_SECRET_KEY are optional. If not specified, Boto3 will handle the configuration for these keys, as described here:

http://boto3.readthedocs.io/en/latest/guide/configuration.html

Optionally, the following environment variables may be set:

  • REDIS_URL="redis://"
  • AWS_CLOUDWATCH_REGION="eu-west-1"

The CloudWatch log group does not have to exist. If it doesn't exists, it will be created.

Running

$ celery cloudwatch

This requires you to have ran setup.py or installed this package through pip. Alternatively, run:

$ python -m celery_cloudwatch

How it works

Celery CloudWatch connects to your broker and monitors tasks in real time. When a task succeeded, failed was rejected or revoked, it uploads all available information about that task into a log stream on AWS CloudWatch Logs.

Based on the specified log group name in the AWS_CLOUDWATCH_GROUP_NAME, a log group will be created. For each possible result, a separate log stream is created. For each task result, an entry is added to the log stream associated with the result.

Known issues

  • --broker on celery cloudwatch is ignored.
  • No descriptive way to specify other brokers than Redis.

All brokers supported by Celery will work, simply specify the broker URL through the REDIS_URL environment variable.