Celery Docker Template

Template to build Celery Docker distributed systems.

┌──────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                      │
│ Docker                              ┌─────────┐                                      │
│                                     │         │                                      │
│              ┌──────────────────────┤  Redis  ├───────────────────────┐              │
│              │                      │         │                       │              │
│              │                      └────┬────┘                       │              │
│              │                           │                            │              │
│              │                           │                            │              │
│              │                           │                            │              │
│     ┌────────┴────────┐         ┌────────┴────────┐         ┌─────────┴────────┐     │
│     │                 │         │                 │         │                  │     │
│     │ celery-worker-1 │         │ celery-worker-2 │         │ celery-worker-...│     │
│     │                 │         │                 │         │                  │     │
│     └────────┬────────┘         └────────┬────────┘         └─────────┬────────┘     │
│              │                           │                            │              │
└──────────────┼───────────────────────────┼────────────────────────────┼──────────────┘
               │                           │                            │
               │                           │                            │
               │              ┌────────────┴───────────┐                │
               │              │                        │                │
               └──────────────┤   Shared mount point   ├────────────────┘
                              │                        │
                              └────────────────────────┘

• Prerequisites
• Makefile configuration
• Build
• Deploy
• Shared mount point

Prerequisites

• make : All deployment is provided under Makefile.

sudo apt install -y make 

• docker

Makefile configuration

• CONCURRENCY :

CONCURRENCY=2

Max value is number of available cores on the machine.

For high intensity tasks, it's better to set a lower concurrency value to save memory and cpu allocation for running tasks.

• MOD_PATH :

MOD_PATH=mods/example.py

Module file path with @celery.task who needs to to be deployed.

Build

Build celery worker image.

make build 

Redis instance is needed to build services (host resolution), so this command will first start Redis container if a Redis host is not found.

Celery worker image name is celery-worker-$MOD_NAME with MOD_NAME the module file name deployed.

Deploy

Deploy worker container :

make run 

Shared mount point

Back up and share data through the shared mount point.

On host, shared mount point is set to /tmp/celery/celery-worker-$MOD_NAME

>/tmp$ 
   ├── celery
   │   └── celery-worker-example