Skip to content

Docker Production v1 and Demo

Alexandar83 edited this page Feb 25, 2025 · 24 revisions

This page outlines the Docker-based Production v1 and Demo setups, highlighting their similarities and differences in architecture, service orchestration with Docker Compose, networking, volume management, and deployment processes.

Table of Contents

  1. Overview
  2. Network Diagram
  3. Docker Compose Services
    3.1 The Wall API
    3.2 Redis
    3.3 PostgreSQL
    3.4 Celery Worker 1
    3.5 Celery Worker 2
    3.6 Celery Beat
  4. Dockerfile
    4.1 Base Image
    4.2 Application Image
    4.3 Celery Lightweight Image
  5. Image Lifecycle
    5.1 Image Build and Push to GHCR
    5.2 Pulling Images and Deployment

1. Overview

The prod_v1 and demo environments follow the same containerized architecture as the local development setup, with the key distinction that the API backend runs inside a Docker container instead of on the host machine. While both configurations are nearly identical, prod_v1 is designed for a full-scale deployment with stricter configurations, whereas demo is a lightweight version optimized for quick setup and minimal dependencies.


2 Network Diagram

The diagram below illustrates the interaction between Docker-managed services and the API in the prod_v1 and demo environments:

Docker Network Prod and Demo

View full image

🗒️ While prod_v2 is also included in the diagram, its deployment specifics differ
and are covered separately in the Docker - Production v2 page.


3. Docker Compose Services

The Docker network for the prod_v1 and demo environments is managed using docker-compose-prod-v1.yml and docker-compose-demo.yml, running all services, including the API, inside Docker containers.

Service name Image Container Name Ports Named Volumes
the_wall_api ghcr.io/alexandar83/the-wall-api:latest
(the-wall-api​-demo:latest)
app 8000:8000 shared_sim_logs:
/app/logs
redis redis:7.4.1 redis_data_prod
(redis_data_demo)
6379:6379 redis_data_prod:/data
(redis_data_demo:/data)
postgres postgres:15.3 postgres_data_prod
(postgres_data_demo)
5432:5432 postgres_data_prod:
/var/lib/postgresql/data

(postgres_data_demo:
/var/lib/postgresql/data)
celery_worker_1 ghcr.io/alexandar83/the-wall-api:latest
(the-wall-api​-demo:latest)
celery_worker_1_prod
(celery_worker_1_demo)
- shared_sim_logs:
/app/logs
celery_worker_2 ghcr.io/alexandar83/the-wall-api:latest
(the-wall-api​-demo:latest)
celery_worker_2_prod
(celery_worker_2_demo)
- shared_sim_logs:
/app/logs
celery_beat ghcr.io/alexandar83/celery-lightweight:latest
(celery-lightweight​-demo:latest)
celery_beat_prod
(celery_beat_demo)
- celery_beat_schedule:
/var/lib/celery/data

✔ The shared_sim_logs volume enables unified log management, allowing the API and the Celery workers to store, update, archive, and delete logs in a shared location.

Additional Service Details:

3.1 The Wall API

Serves as the primary application service.

  • prod_v1:

    • Uses the_wall_api_prod.env to load environment variables.
    • Secrets:
      • Loads the Redis password from config/envs/prod/redis_prod.env.
      • Loads the PostgreSQL credentials from config/envs/prod/postgres_prod.env.
      • Loads the app's secret key from config/envs/prod/django_secret_key_prod.env.
    • Sets PROJECT_MODE=prod_v1 to indicate the production version of the environment.

    ✔ Separating secrets into dedicated environment files ensures security and flexibility in production deployments.

  • demo:

    ✔ Managing authentication in Dockerfile.demo removes the need for external environment configuration and streamlines deployment.

  • Common Configuration:

    • The container starts with the following command:

      python manage.py makemigrations the_wall_api &&
      python manage.py migrate &&
      exec gunicorn --bind 0.0.0.0:8000 config.wsgi
      • Command Breakdown:

        • makemigrations ensures any pending model changes are recognized.
        • migrate applies database migrations before starting the server.
        • gunicorn is used as the production WSGI server, binding to all interfaces.

        ✔ Using gunicorn ensures efficient request handling in a production setting, optimizing performance under load.

    • Uses the depends_on service_healthy condition to wait until PostgreSQL passes its health check.

      ✔ This health check guarantees that the database is fully initialized before migrations are applied and the API attempts to connect.

3.2 Redis

Caches data and serves as a message broker for Celery task queues.

  • prod_v1:

    • Uses a Docker-managed volume (redis_data_prod) to persist data across restarts
    • Mounts config/envs/common/redis.conf to /usr/local/etc/redis/redis.conf inside the container.
    • The start command incorporates a password using:
      redis-server /usr/local/etc/redis/redis.conf --requirepass "$$REDIS_PASSWORD"
      • The password is loaded from config/envs/prod/redis_prod.env.

    ✔ Binding redis.conf allows dynamic configuration without altering the base image, enabling environment-specific tuning without container rebuilds.

  • demo:

    • Persists data using a Docker-managed volume (redis_data_demo).
    • Uses a start command with hardcoded settings, replicating the configuration found in config/envs/common/redis.conf.
    • Does not use a password.

    ✔ Hardcoding settings inside the start command and omitting password configuration simplifies the demo deployment, avoiding the need for external configuration management.

💡 For a breakdown of the settings in redis.conf, see the redis configuration file section.

3.3 PostgreSQL

Provides persistent database storage for the API.

  • prod_v1:

    • Uses a Docker-managed volume (postgres_data_prod) to preserve data across restarts.
    • Credentials are loaded from config/envs/prod/postgres_prod.env, which defines:

    ✔ Loading credentials from postgres_prod.env decouples secrets from the container, enabling secure and configurable deployments.

  • demo

    • Retains data using a Docker-managed volume (postgres_data_demo).
    • Authentication is hardcoded in the docker-compose-demo.yml file:
      - POSTGRES_DB=db_name_demo
      - POSTGRES_USER=db_user_demo
      - POSTGRES_PASSWORD=db_password_demo

    ✔ Hardcoding authentication ensures a quick setup without requiring external environment configuration.

  • Common Configuration:

    • Includes a health check that runs pg_isready to ensure the database is ready before dependent services start.

    ✔ The built-in health check (pg_isready) ensures that The Wall API service starts once PostgreSQL is fully initialized, preventing connection failures.

3.4 Celery Worker 1

Executes sequential Celery tasks.

  • prod_v1:

  • demo:

  • Common Configuration:

    • Uses the same image as The Wall API service.

      • Starts with the start_celery_computation_worker.sh script which:
      • Runs the Celery worker without a --concurrency flag - concurrency is adjusted automatically based on available system resources.
      • Subscribes the worker to the concurrent_tasks queue.
    • Includes a health check that verifies Celery is running by executing celery -A config status.

3.5 Celery Worker 2

Executes sequential Celery tasks.

  • In both the prod_v1 and demo environments, Worker 2 uses an identical configuration structure to Celery Worker 1, except for the starting command:
    celery -A config worker --loglevel=info --queues=sequential_tasks --concurrency=1
    • Command Breakdown:
      • celery -A config worker: Start a Celery worker, using the Celery app instance from the config/celery.py module.
      • --loglevel=info: Set the log level inside the worker process to info.
      • --queues=sequential_tasks: Subscribe to the sequential_tasks queue.
      • --concurrency=1: Set the concurrency to 1 to ensure sequential task execution.

3.6 Celery Beat

Schedules periodic Celery tasks such as DB cleanup, File archiving, and Archive cleanup.

  • prod_v1:

  • demo:

  • Common Configuration:

    • Start command details:
      • -A config beat: Run Celery Beat.
      • -s /var/lib/celery/data/celerybeat-schedule: Set the schedule file location inside the container.
    • Stores the task schedule persistently in the celery_beat_schedule volume, to prevent task loss across container restarts.
    • Uses lightweight Celery configuration (LIGHT_CELERY_CONFIG=True), preventing unnecessary Django initialization.
    • Depends on Celery Worker 1 and Celery Worker 2, ensuring they are ready before starting.

    ✔ The lightweight image reduces overhead by excluding unnecessary dependencies.


4. Dockerfile

Dockerfile.prod and Dockerfile.demo define multi-stage builds, which are used in Docker Compose
to create the images for the API, Celery Worker 1, Celery Worker 2, and Celery Beat.

4.1 Base Image

Defines the foundation for the next build stages.

  • Common Configuration:

    • Sets the environment variables:

      • PYTHONDONTWRITEBYTECODE=1 (disables .pyc file generation).
      • PYTHONUNBUFFERED=1 (ensures real-time logging output).
      • PYTHON_VERSION=3.11.9 (sets a reference environment variable, useful for scripts and automation, but not affecting the installed Python version).
      • LOGS_DIR_NAME=logs (centralizes the logs folder name for use across all build stages, ensuring consistency for the application and Celery workers).
    • Sets the working directory to /app.

    • Creates a non-root user for security:

      RUN adduser -u 5678 --disabled-password --gecos "" appuser
      • User password is disabled to skip interactive login.
      • User information is left empty to avoid prompts.
      • The user is named 'appuser'.
  • demo:

    • Loads additional environment variables specific to the demo environment:
      • Preconfigured Redis connection URL.
      • Pre-defined PostgreSQL authentication details.
      • Pre-defined Django secret key.
      • Pre-defined API version.

    ✔ Embedding essential environment variables allows demo to run independently of external configuration files, simplifying quick deployments.

✔ Using a minimal slim-bullseye base image reduces container size and security vulnerabilities.

✔ The non-root user enforces security best practices by restricting elevated privileges within the container.

4.2 Application Image

Used for building Celery Worker 1 and Celery Worker 2, which require access to the full application codebase for task execution, as well as for the API container.

  • prod_v1:

    • Target: the_wall_api_image
  • demo:

    • Target: the_wall_api_demo_image
  • Common Configuration:

    • Installs system dependencies gcc and libpq-dev (required to compile the psycopg2 PostgreSQL adapter from source).
    • Copies requirements.txt to the container and installs Python dependencies using pip.
    • Additionally installs gunicorn to serve the application as a WSGI server.
      🗒️ gunicorn is not listed in requirements.txt because it is not an application dependency.
    • Copies the entire application to /app inside the container.
      🗒️ .dockerignore controls which files are copied, excluding unnecessary files to optimize the image size and build process.
    • Sets up log folder permissions:
      • Creates /app/${LOGS_DIR_NAME} to prevent permission issues for Celery workers.
      • Assigns ownership to the non-root user (appuser).
    • Runs subsequent processes as the non-root appuser.

✔ Image size is reduced by:
  • Deleting system package manager caches after installing PostgreSQL dependencies (rm -rf /var/lib/apt/lists/)
  • Preventing package caching during Python dependency installation (--no-cache-dir)

4.3 Celery Lightweight Image

Used in the Celery Beat service.

  • prod_v1:

    • Target: celery_lightweight_image
  • demo:

    • Target: celery_lightweight_demo_image
  • Common Configuration:

    • Extracts and installs only Celery and Redis dependencies from requirements.txt using:

      RUN grep -E '^celery==|^redis==' /app/requirements.txt > filtered-requirements.txt
      RUN pip install --no-cache-dir -r filtered-requirements.txt

      ✔ The extraction command guarantees that Celery and Redis are installed with the exact versions specified in requirements.txt, ensuring consistency across deployments.

    • Copies the full application to /app.
      🗒️ Controlled via .dockerignore.

    • Sets up Celery Beat schedule and logs storage:

      • Creates /var/lib/celery/data to persist the task schedule.
      • Creates the logs folder and assigns permissions to appuser.

✔ This strategy shrinks the package footprint, enhancing both container compactness and build speed.


5. Image Lifecycle

Docker images for the prod_v1 and demo environments follow a structured build, push, and deployment process. A full application image supports the API and Celery workers, while a lightweight Celery image is optimized for Celery Beat. Both images are tagged and stored in GHCR for versioned and centralized distribution.

📝 All console commands in this section should be executed from the app's root folder.

5.1 Image Build and Push to GHCR

  • prod_v1:

    Full Image
    docker build --no-cache \
    --target the_wall_api_image \
    -t ghcr.io/alexandar83/the-wall-api:v2 \
    -t ghcr.io/alexandar83/the-wall-api:latest \
    -f ./config/docker/Dockerfile.prod \
    .
    
    docker push ghcr.io/alexandar83/the-wall-api --all-tags
    • Commands Breakdown
      • --no-cache: Ensures a clean build without cached layers.
      • --target the_wall_api_image: Specifies the multi-stage build target for the API and Celery workers image.
      • -t ghcr.io/alexandar83/the-wall-api:v2 -t ghcr.io/alexandar83/the-wall-api:latest: Creates two tags for the built image - v2 and latest.
      • -f ./config/docker/Dockerfile.prod: Specifies the path to the Dockerfile to use for the build.
      • .: Specifies the current folder as the context directory for the build.

      • docker push ghcr.io/alexandar83/the-wall-api --all-tags: Pushes all image tags to GHCR.

    Lightweight Image
    docker build --no-cache \
    --target celery_lightweight_image \
    -t ghcr.io/alexandar83/celery-lightweight:v2 \
    -t ghcr.io/alexandar83/celery-lightweight:latest \
    -f ./config/docker/Dockerfile.prod \
    .
    
    docker push ghcr.io/alexandar83/celery-lightweight --all-tags

  • demo: The commands are replicated from the prod_v1 workflow.

    Full Image
    docker build --no-cache \
    --target the_wall_api_demo_image \
    -t ghcr.io/alexandar83/the-wall-api-demo:v2 \
    -t ghcr.io/alexandar83/the-wall-api-demo:latest \
    -f ./config/docker/Dockerfile.demo \
    .
    
    docker push ghcr.io/alexandar83/the-wall-api-demo --all-tags

    Lightweight Image
    docker build --no-cache \
    --target celery_lightweight_demo_image \
    -t ghcr.io/alexandar83/celery-lightweight-demo:v2 \
    -t ghcr.io/alexandar83/celery-lightweight-demo:latest \
    -f ./config/docker/Dockerfile.demo \
    .
    
    docker push ghcr.io/alexandar83/celery-lightweight-demo --all-tags

5.2 Pulling Images and Deployment

  • prod_v1:

    docker compose -p the-wall-api-prod -f config/docker/docker-compose-prod-v1.yml up -d
    • Command Breakdown:
      • -p the-wall-api-prod: Assigns a name to the Docker Compose project (affects network names, container names, and volumes).
      • -f config/docker/docker-compose-prod-v1.yml: Specifies the path to the Docker Compose file.
      • up: Creates the containers defined in the Compose file and starts them.
      • -d: Runs the containers in the background, allowing the console shell in which the command is executed to be used for other tasks.

  • demo:

    docker compose -p the-wall-api-demo -f docker-compose-demo.yml up -d
    • The command should be run from the folder where the docker-compose-demo.yml file is located or with an absolute path.



💡 For detailed environment setup instructions, see the Production Setup v1 and Quick App Setup guides.


Next:   Docker - Production v2
Back to: Docker - Local Development

Return to Top
DTS Index
Home Page


Clone this wiki locally