FastAPI + PostgresSQL + Celery + Redis Example

An example demonstrates how to perform heavy and long-time background computation tasks(such as running a machine learning model) in microservices.

This repo was built up with two services: API service and Worker service.

Service tech stack:

• FastAPI (api server)
• Celery (worker)
• PostgresSQL (persist task in the database)
• Redis (message broker)

API Service

• REST API
  • create task
  • query task
• Message Queue
  • create task
  • query task Worker Service
• Message Queue
  • run task
  • send task progress

Workflow:

NOTE:

• Task is defined by a custom FastAPI App frontend user.
• Celery Task is defined by Celery for internal
  • task model
  • result backend
• Update Celery Task progress with custom states
  • celery task getting progress · GitHub
  • python - How to retrieve meta from celery backend storage - Stack Overflow

Get Started

docker run -it --rm fastapi-celery-web bash

docker exec -it fastapi-celery-db-1 bash

docker-compose build
# CTRL+C: just stop containers while keeping data
docker-compose up
docker-compose up -d

# Remove containers with wiping data
docker-compose down 

NOTE:

• Dockerfile.dev in app folder is used for development in a container environment. It builds a virtual environment with all dependencies installed to let you use such a hot reload feature when coding.