Docker Local Development

This page details the Docker-based local development setup, covering the architecture, service orchestration with Docker Compose, networking, volume management, and startup configurations.

Table of Contents

1. Overview
2. Network Diagram
3. Docker Compose Services
   3.1 Redis
   3.2 PostgreSQL
   3.3 Wait for PostgreSQL
   3.4 Celery Worker 1
   3.5 Celery Worker 2
   3.6 Celery Beat
4. Dockerfile
   4.1 Base Image
   4.2 Application Image
   4.3 Celery Lightweight Image
5. Startup Script

1. Overview

The Docker - Local Development setup runs the API on the host machine, with PostgreSQL, Redis, and Celery containerized. This ensures a smooth workflow with real-time code updates while mirroring production. Docker Compose manages service orchestration, optimizing networking, environment variables, and startup scripts for efficient development.

---

2. Network Diagram

The following diagram visualizes how Docker-managed services interact with the API in the local development setup:

View full image

Return to Top

---

3. Docker Compose Services

The Docker network for the dev environment uses docker-compose-dev.yml to manage supporting services while keeping the API running on the local host.

• Core Services

  Service Name	Image	Container Name	Ports	Named Volumes
  redis	redis:7.4.1	redis_data_dev	6379:6379	redis_data_dev: /data
  postgres	postgres:15.3	postgres_data_dev	5432:5432	postgres_data_dev: /var/lib/postgresql/data
  wait_for_postgres	busybox:1.36.1	-	-	-
  celery_worker_1	ghcr.io/alexandar83/ the-wall-api:latest	celery_worker_1_dev	-	-
  celery_worker_2	ghcr.io/alexandar83/ the-wall-api:latest	celery_worker_2_dev	-	-
  celery_beat	ghcr.io/alexandar83/ celery-lightweight:latest	celery_beat_dev	-	celery_beat_schedule: /var/lib/celery/data

• Bind Mounts

  Service Name	Mounted Paths (Host → Container)
  redis	config/envs/common/redis.conf → /usr/local/etc/redis/redis.conf
  celery_worker_1	Host app root folder → /app
  celery_worker_2	Host app root folder → /app

  ✔ The application root is configured as a bind mount, allowing live code updates without image rebuilds during development.
  ✔ Logs are centralized in the logs/ subfolder ensuring shared access to storage, updates, archiving, and deletion for the application and the Celery workers.
  
  

Additional Service Details:

3.1 Redis

Caches data and serves as a message broker for Celery task queues.

• Uses a Docker-managed volume (redis_data_dev) to persist data across restarts.
• Loads its configuration from redis.conf located in config/envs/common/.
• Mounts config/envs/common/redis.conf to /usr/local/etc/redis/redis.conf inside the container.
• Starts using the command redis-server /usr/local/etc/redis/redis.conf.

💡 For a breakdown of redis.conf settings, see the redis configuration file section.

✔ Using a bind mount for redis.conf ensures configurability without modifying the base image, enabling seamless updates without rebuilding the container.

Return to Top

3.2 PostgreSQL

Provides persistent database storage.

• Uses a Docker-managed volume (postgres_data_dev) to preserve data across restarts.
• Uses config/envs/dev/postgres_dev.env to load environment variables for authentication (POSTGRES_DB, POSTGRES_PASSWORD, and POSTGRES_USER).
• Includes a health check that runs pg_isready to ensure availability before dependent services start.

✔ The health check allows the Wait for Postgres service to reliably detect when PostgreSQL is ready.

Return to Top

3.3 Wait for Postgres

Ensures the PostgreSQL service is fully available before Celery services start.

• Runs a lightweight busybox container.
• Uses the depends_on service_healthy condition to wait until PostgreSQL passes its health check.

✔ Used to facilitate the correct execution of the docker-compose-dev+migrations.sh deployment script
by ensuring the PostgreSQL service is fully ready before migrations are applied.

Return to Top

3.4 Celery Worker 1

Executes concurrent Celery tasks.

• Builds from the multi-stage Dockerfile.dev using the_wall_api_image target.

  • Build context: Uses the Host app root folder (context: ../..), ensuring access to the full repository.

  • Build arguments - exported as environment variables in the docker-compose-dev+migrations.sh script:

    • LOGS_DIR_NAME
    • APP_USER_ID
    • APP_GROUP_ID
    • APP_GROUP_NAME

    🗒️ Used to configure the permissions of the non-root user in the container.

• Starts with the start_celery_computation_worker.sh script which:

  • Runs the Celery worker without a --concurrency flag - concurrency is adjusted automatically based on available system resources.
    • If CONCURRENT_SIMULATION_MODE is set to multiprocessing_v1,2, or 3, uses explicit concurrency config (see script for details).
  • Subscribes the worker to the concurrent_tasks queue.

• Mounts the host application root folder (../..) into the container's /app directory

• Uses the_wall_api_dev.env to load environment variables.

• Sets the environment variable STARTED_FROM_CELERY_SERVICE=True to indicate it is running as a Celery service.

• Includes a health check that verifies Celery is running by executing celery -A config status.

🗒️ Note: Docker Compose resolves relative paths based on the location of the Compose file (config/docker/docker-compose-dev.yml), making the ../.. path referring to the root of the repository.

✔ The application root is configured as a bind mount, allowing live code updates without the need for image rebuilds during development.

Return to Top

3.5 Celery Worker 2

Executes sequential Celery tasks.

• Uses the image built for the Celery Worker 1 service.
• Starting command celery -A config worker --loglevel=info --queues=sequential_tasks --concurrency=1 breakdown:
  • celery -A config worker: Start a Celery worker, using the Celery app instance from the config/celery.py module.
  • --loglevel=info: Set the log level inside the worker process to info.
  • --queues=sequential_tasks: Subscribe to the sequential_tasks queue.
  • --concurrency=1: Set the concurrency to 1 to ensure sequential task execution.
    
    
• Configurations identical to Celery Worker 1:
  • Host app root folder mount.
  • Environment variables load from the_wall_api_dev.env.
  • STARTED_FROM_CELERY_SERVICE environment variable.
  • Health check.

✔ The pre-built image for Worker 1 is reused, ensuring consistency and reducing build time by avoiding redundant image creation.

Return to Top

3.6 Celery Beat

Schedules periodic Celery tasks such as DB cleanup, File archiving, and Archive cleanup.

• Builds from Dockerfile.dev using the celery_lightweight_image target.

• Start command details:

  • -A config beat: Run Celery Beat.
  • -s /var/lib/celery/data/celerybeat-schedule: Set the schedule file location inside the container.
    ✔ Celery Beat’s schedule file is placed outside /app to prevent unintended local creation in the bind-mounted folder, ensuring proper isolation.

• Stores the task schedule persistently in the celery_beat_schedule volume, to prevent task loss across container restarts.

• Uses lightweight Celery configuration (LIGHT_CELERY_CONFIG=True), preventing unnecessary Django initialization.

• Depends on Celery Worker 1 and Celery Worker 2, ensuring they are ready before starting.

• Configurations identical to Celery Worker 1:
  • Build context and arguments.
  • Host app root folder mount.
  • Environment variables load from the_wall_api_dev.env.
  • STARTED_FROM_CELERY_SERVICE environment variable.

✔ The lightweight image reduces overhead by excluding unnecessary dependencies.

Return to Top

---

4. Dockerfile

The Dockerfile.dev defines a multi-stage build, used in Docker Compose
to create the Celery images for Celery Worker 1, Celery Worker 2, and Celery Beat.

4.1 Base Image

Provides the base for the next two stages.

• Uses python:3.11.9-slim-bullseye as the base image.

• Sets the environment variables:

  • PYTHONDONTWRITEBYTECODE=1 (disables .pyc file generation).
  • PYTHONUNBUFFERED=1 (ensures real-time logging output).
  • PYTHON_VERSION=3.11.9 (sets a reference environment variable, useful for scripts and automation, but not affecting the installed Python version).

• Sets the working directory to /app.

• Defines build arguments, passed as ARG values from the docker-compose-dev.yml file:

  • LOGS_DIR_NAME
  • APP_USER_ID
  • APP_GROUP_ID
  • APP_GROUP_NAME

• Creates a non-root user and a dedicated group:

  • The group and user are created dynamically using the provided build arguments:

    RUN groupadd --gid ${APP_GROUP_ID} ${APP_GROUP_NAME} && \
    adduser --uid ${APP_USER_ID} --gid ${APP_GROUP_ID} --disabled-login --gecos "" appuser

    • The user is assigned to the group.
    • User password is disabled to skip interactive login.
    • User information is left empty to avoid prompts.
    • The user is named 'appuser'.

  ✔ The APP_GROUP_ID and APP_GROUP_NAME build arguments are required on Linux to set user and group permissions for proper volume access. While they have no effect on Windows, they are still passed to maintain a consistent build process without introducing platform-specific handling.

  💡 See the 3.9 Linux Development Configuration Variables section of the Environment Variables Index for details on the dedicated group's usage.
  
  

✔ A minimal slim-bullseye base image is used to reduce container size and attack surface.

✔ Runtime efficiency is optimized by preventing unnecessary (.pyc) files generation.

Return to Top

4.2 Application Image

Used in Celery Worker 1 and Celery Worker 2, which require access to the full application codebase for task execution.

• Target: the_wall_api_image.
• Installs system dependencies gcc and libpq-dev (required to compile the psycopg2 PostgreSQL adapter from source).
• Copies requirements.txt to the container and installs Python dependencies using pip.
• Copies the entire application to /app inside the container.
• Sets up log folder permissions:
  • Creates /app/${LOGS_DIR_NAME} to prevent permission issues for Celery workers.
  • Assigns ownership to the non-root user (appuser).
• Runs subsequent processes as the non-root appuser.

✔ Image size is optimized by:
  • Removing system package manager caches after installing PostgreSQL dependencies (rm -rf /var/lib/apt/lists/)
  • Disabling package caching during Python dependency installation (--no-cache-dir)

Return to Top

4.3 Celery Lightweight Image

Used in the Celery Beat service.

• Target: celery_lightweight_image

• Extracts and installs only Celery and Redis dependencies from requirements.txt using:

  RUN grep -E '^celery==|^redis==' /app/requirements.txt > filtered-requirements.txt
  RUN pip install --no-cache-dir -r filtered-requirements.txt

  ✔ The extraction command enforces the use of exact dependency versions from requirements.txt, ensuring uniformity across deployments.
  
  

• Copies the full application to /app in the container.

• Sets up Celery Beat schedule and log storage:

  • Creates /var/lib/celery/data to persist the task schedule.
  • Creates the logs folder and assigns permissions to appuser.
  
  

✔ This approach minimizes the package footprint, reducing both container size and build time.

Return to Top

---

5. Startup Script

The docker-compose-dev+migrations.sh script is used to initialize the development environment.

Stages:

1. Launch Redis and PostgreSQL services.
2. Wait for the Wait for Postgres service to finish.
3. Apply database migrations.
4. Configure logs folder.
5. Build Celery services.
6. Start Celery services.

💡 See docker-compose-dev+migrations.sh for a detailed script description.

💡 For in-depth environment setup instructions, see the Windows and Linux (WSL2) guides for dev deployment.

---

Next:   Docker - Production v1 and Demo
Back to: Containerization

Return to Top
DTS Index
Home Page

---