Skip to content

Deployment Guide

Jump to bottom
Ömer Tarık Yılmaz edited this page Apr 5, 2026 · 1 revision

Deployment Guide

This guide covers deploying White-Ops in production, from single-machine setups to multi-PC worker fleets.

Table of Contents

Single Machine Setup

The simplest deployment runs all services on one machine.

Minimum Requirements

Resource Minimum Recommended
CPU 4 cores 8+ cores
RAM 8 GB 16+ GB
Disk 20 GB 100+ GB SSD
OS Ubuntu 22.04 / Debian 12 Ubuntu 24.04

Steps

# 1. Install Docker
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
# Log out and back in

# 2. Clone the repository
git clone https://github.com/your-org/white-ops.git
cd white-ops

# 3. Configure environment
cp .env.example .env
# Edit .env with production values (see Environment Variables below)

# 4. Build and start
make build
make up

# 5. Run database migrations
make migrate

# 6. Verify
make status
curl http://localhost:8000/health

The admin panel is available at http://your-server-ip:3000.

Multi-PC Fleet

For scaling, run additional workers on separate machines. The master (server, database, web) runs on one machine while workers run on any number of others.

Master Machine

Deploy the full stack as described in Single Machine Setup.

Ensure these ports are accessible from worker machines:

  • 8000 -- API server (workers connect here)
  • 6379 -- Redis (workers need access)
  • 9000 -- MinIO (workers store/retrieve files)
  • 8025 -- Mail server (workers send/receive mail)

Worker Machines

Use the provided setup script:

# On each worker machine:
./scripts/add-worker.sh <master-ip>

The script:

  1. Installs Docker if not present
  2. Tests connectivity to the master
  3. Creates a docker-compose.yml for the worker
  4. Outputs next steps

After running the script:

cd ~/white-ops-worker

# Create .env with credentials matching the master
cat > .env << EOF
REDIS_PASSWORD=same-as-master
MINIO_ROOT_USER=whiteops
MINIO_ROOT_PASSWORD=same-as-master
ANTHROPIC_API_KEY=sk-ant-...
DEFAULT_LLM_PROVIDER=anthropic
DEFAULT_LLM_MODEL=claude-sonnet-4-20250514
EOF

# Start the worker
docker compose up -d

Then approve the worker in the admin panel at System > Workers.

See Worker-Management for detailed fleet management.

Docker Compose Configuration

Production (docker-compose.yml)

The main compose file defines all services:

Service Image Ports Resources
postgres postgres:16-alpine 5432 --
redis redis:7-alpine 6379 --
minio minio/minio:latest 9000, 9001 --
server Built from ./server 8000 2 CPU, 1 GB RAM
worker Built from ./worker -- --
web Built from ./web 3000 --
mail Built from ./mail 8025 --

Optional (monitoring profile):

Service Image Ports
prometheus prom/prometheus:latest 9090
grafana grafana/grafana:latest 3001

Development (docker-compose.dev.yml)

Overrides for development with hot reload, debug logging, and volume mounts.

make dev

Scaling Workers Locally

To run multiple workers on the same machine:

docker compose up -d --scale worker=3

Each worker registers independently with a unique name.

Environment Variables

See Configuration for the complete reference table. The critical production variables are:

Variable Description Required
SECRET_KEY Application secret (32+ chars) Yes
JWT_SECRET_KEY JWT signing secret (32+ chars) Yes
POSTGRES_PASSWORD Database password Yes
REDIS_PASSWORD Redis password Yes
MINIO_ROOT_PASSWORD MinIO password (8+ chars) Yes
ADMIN_PASSWORD Initial admin login password Yes
ANTHROPIC_API_KEY Anthropic API key (if using Claude) At least one LLM key
OPENAI_API_KEY OpenAI API key (if using GPT) At least one LLM key
DEFAULT_LLM_PROVIDER anthropic, openai, google, ollama Yes
DEFAULT_LLM_MODEL Model identifier Yes

Generating Secure Secrets

# Generate a random 64-character string
openssl rand -hex 32

# Or use Python
python3 -c "import secrets; print(secrets.token_urlsafe(48))"

Production Checklist

Run the pre-deployment check script:

make check

Manual checklist:

  • Secrets: All passwords and keys are unique, random strings (not defaults)
  • DEBUG: Set DEBUG=false
  • APP_ENV: Set APP_ENV=production
  • CORS: CORS_ORIGINS set to your actual frontend domain
  • TLS: HTTPS configured via reverse proxy (see below)
  • Firewall: Only expose necessary ports (3000, 8000 or just 443 via proxy)
  • Backups: Database backup schedule configured
  • Monitoring: Prometheus + Grafana enabled (make monitoring)
  • Log Rotation: Docker log rotation configured
  • Resource Limits: Docker resource constraints set for each service
  • LLM Keys: At least one LLM provider API key configured
  • Mail: SMTP configured if external email is needed
  • DNS: Domain name pointing to the server
  • Workers: All workers approved in admin panel

Docker Log Rotation

Add to /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  }
}

Then restart Docker: sudo systemctl restart docker

TLS / HTTPS

White-Ops does not handle TLS directly. Use a reverse proxy.

Nginx Example

server {
    listen 443 ssl http2;
    server_name whiteops.example.com;

    ssl_certificate /etc/letsencrypt/live/whiteops.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/whiteops.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    # Admin panel
    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # API server
    location /api/ {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # WebSocket
    location /ws {
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_read_timeout 86400;
    }
}

server {
    listen 80;
    server_name whiteops.example.com;
    return 301 https://$host$request_uri;
}

Let's Encrypt

sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d whiteops.example.com

Update CORS_ORIGINS in .env to match your HTTPS domain.

Backup and Restore

Database Backup

# Manual backup
make backup
# Saves to ./backups/whiteops_YYYYMMDD_HHMMSS.sql

# Restore from backup
make restore FILE=backups/whiteops_20250101_120000.sql

Automated Backups

Add a cron job on the host:

# Daily backup at 2am, keep 30 days
0 2 * * * cd /opt/white-ops && make backup && find backups/ -mtime +30 -delete

MinIO File Backup

MinIO data is stored in a Docker volume. Back it up with:

# Export volume
docker run --rm -v white-ops_minio_data:/data -v $(pwd)/backups:/backup \
  alpine tar czf /backup/minio_$(date +%Y%m%d).tar.gz -C /data .

# Restore volume
docker run --rm -v white-ops_minio_data:/data -v $(pwd)/backups:/backup \
  alpine tar xzf /backup/minio_20250101.tar.gz -C /data

Full System Backup

# Stop services, backup everything
make down
tar czf white-ops-full-backup-$(date +%Y%m%d).tar.gz \
  backups/ .env docker-compose.yml
make up

Monitoring Setup

Enable Prometheus and Grafana

make monitoring

This starts:

  • Prometheus at http://localhost:9090 -- metrics collection
  • Grafana at http://localhost:3001 -- dashboards

Grafana Access

Default credentials:

  • Username: admin
  • Password: value of GRAFANA_ADMIN_PASSWORD in .env

Pre-configured dashboards are provisioned automatically from monitoring/grafana/dashboards/.

Prometheus Configuration

The Prometheus config is at monitoring/prometheus/prometheus.yml. It scrapes:

  • The API server /metrics endpoint
  • Worker metrics
  • PostgreSQL exporter (if configured)
  • Redis exporter (if configured)

Alerting

Configure Prometheus alerting rules in monitoring/prometheus/ and set up Grafana alert channels (email, Slack, webhook) for critical events:

  • Worker offline for > 5 minutes
  • Task failure rate > 10%
  • CPU/RAM usage > 90%
  • Database connection pool exhausted
  • API response time > 5 seconds

Troubleshooting

Services Not Starting

# Check logs for a specific service
make logs-server
make logs-worker
docker compose logs postgres
docker compose logs redis

# Check health status
make status

Database Connection Refused

  1. Verify PostgreSQL is healthy: docker compose ps postgres
  2. Check credentials match between .env and the server
  3. Ensure the database volume exists: docker volume ls | grep postgres
  4. Try restarting: docker compose restart postgres

Worker Not Connecting

  1. Check master is reachable from worker: curl http://<master-ip>:8000/health
  2. Verify MASTER_URL in worker environment
  3. Check worker logs: docker compose logs worker
  4. Ensure worker is approved in admin panel
  5. Verify Redis and MinIO are accessible from the worker

High Memory Usage

  1. Check which service is consuming memory: docker stats
  2. Review resource limits in docker-compose.yml
  3. Consider scaling workers across multiple machines
  4. Check for tasks stuck in in_progress state

Task Stuck in Pending

  1. Verify at least one worker is online and approved
  2. Check that an agent is assigned and in active state
  3. Review server logs for assignment errors
  4. Check Redis connectivity

LLM API Errors

  1. Verify API key is correct in .env
  2. Check API key has sufficient quota/credits
  3. Test key directly: curl https://api.anthropic.com/v1/messages -H "x-api-key: $ANTHROPIC_API_KEY" ...
  4. Check worker logs for specific error messages
  5. Try switching to a different LLM provider

Performance Optimization

  • Increase WORKER_MAX_AGENTS if workers have spare CPU/RAM
  • Add more workers for horizontal scaling
  • Increase PostgreSQL shared_buffers and work_mem for large datasets
  • Enable Redis persistence if task queue reliability is critical
  • Use SSDs for database and MinIO storage

Clone this wiki locally