Skip to content

CI CD

Jump to bottom
Valerio edited this page Apr 22, 2026 · 5 revisions

CI/CD and Production Deployment

Harpyx ships with a GitHub Actions pipeline that builds the two application images and pushes them to the GitHub Container Registry (ghcr.io).

The pipeline publishes only the Harpyx application images:

  • ghcr.io/<owner>/harpyx-webapp
  • ghcr.io/<owner>/harpyx-worker

Backing services such as SQL Server, MinIO, RabbitMQ, OpenSearch, Redis, and ClamAV are not built or published by this pipeline. They may be run as containers, provided directly by the host, or consumed as centralized/shared services depending on the production topology you choose.

Pipeline overview

The workflow is defined in .github/workflows/deploy.yml and triggers on every push to main (or manually from the Actions tab).

Developer machine
      |  git push -> main
      v
GitHub Actions
  |- docker build -> ghcr.io/<owner>/harpyx-webapp:latest
  |- docker build -> ghcr.io/<owner>/harpyx-worker:latest
  `- images are ready for deployment

Deployment to the production server can then be handled in one of two ways:

  • directly by GitHub Actions over SSH
  • manually on the server after the images have been published

Why two services

Harpyx is deployed as two cooperating application processes:

  • webapp serves the HTTP UI/API, handles authentication, uploads, and user-facing requests
  • worker consumes background jobs from RabbitMQ and executes the slow asynchronous work that should not block HTTP requests

In practice, the Worker is responsible for document ingestion tasks such as parsing, extraction, chunking, embedding, indexing, retry handling, and scheduled cleanup jobs. That is why production deployments normally update both webapp and worker together, even though only webapp receives direct browser traffic.

For the deeper architectural view, see Architecture and Jobs and Messaging.

Key files

File Purpose
.github/workflows/deploy.yml GitHub Actions workflow: build and push the application images to GHCR
docker-compose.prod.example.yml Example production Compose file. It can be used as a starting point for an all-container deployment, then adapted to the real production topology
docker-compose.yml Development Compose file: builds images locally and exposes the full local stack
infra/Dockerfile.webapp Multi-stage build for Harpyx.WebApp
infra/Dockerfile.worker Multi-stage build for Harpyx.Worker
.env.example Template for environment variables: copy to .env, never commit the real .env

Production topology

docker-compose.prod.example.yml is intentionally a convenient baseline, not a mandatory final shape.

You can use it almost as-is if you want to run all backing services as containers on the same host. You can also modify it easily if some services are centralized or managed elsewhere.

Typical adaptations include:

  • removing containerized services that are provided externally, such as SQL Server
  • changing Database__Host, Database__Port, Minio__Endpoint, RabbitMQ__HostName, Redis__ConnectionString, or OpenSearch__Endpoint to point to shared infrastructure
  • keeping only webapp, worker, and the backing containers that this specific deployment should own

For example, if production uses a centralized SQL Server, the real docker-compose.prod.yml can omit the sqlserver service entirely, while webapp and worker connect to the external database through .env.

Ubuntu production deployment

This section describes a generic Ubuntu-based deployment flow. Exact details depend on whether your backing services are fully containerized or partly centralized.

Prerequisites

  • Ubuntu 22.04 LTS or later
  • Docker Engine installed from the official Docker repository
  • A domain name pointed at the server IP
  • A GitHub account with write access to the repository

If the docker-compose-plugin package is unavailable from your current apt sources, install the standalone Compose v2 binary:

sudo curl -SL https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64 \
  -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose version

Step 1 - SSH key pair for GitHub Actions

Generate a dedicated key pair on your development machine. You can leave the passphrase empty for simplicity, or protect the key with a passphrase and store it in the SSH_PASSPHRASE GitHub secret.

# Windows PowerShell
ssh-keygen -t ed25519 -C "github-actions-deploy" -f "$env:USERPROFILE\.ssh\harpyx_deploy"
# macOS / Linux
ssh-keygen -t ed25519 -C "github-actions-deploy" -f ~/.ssh/harpyx_deploy

Authorize the public key on the server:

# Windows PowerShell
type "$env:USERPROFILE\.ssh\harpyx_deploy.pub" | ssh user@your-server "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
# macOS / Linux
ssh-copy-id -i ~/.ssh/harpyx_deploy.pub user@your-server

Step 2 - GitHub repository secrets

Go to https://github.com/Ryadel/Harpyx/settings/secrets/actions and add the repository secrets required by your deploy workflow.

Typical values are:

Secret Value
SSH_HOST Server IP address or hostname
SSH_USER SSH username
SSH_KEY Full content of the deploy private key
SSH_PASSPHRASE Passphrase for the private key, if the key is encrypted
SSH_PORT SSH port, usually 22
DEPLOY_PATH Absolute path on the server, for example /var/www/harpyx

GITHUB_TOKEN is provisioned automatically by GitHub Actions and is used by the workflow to push images to GHCR.

Step 3 - Container registry visibility

After the first successful Actions run, the packages harpyx-webapp and harpyx-worker appear under the GitHub Packages area for the repository owner.

If those packages remain private, the production server must authenticate to ghcr.io before pulling them. If you prefer public pulls, you can make the packages public instead.

Step 4 - Server directory and environment file

Create the deployment directory and write the production .env file:

mkdir -p /var/www/harpyx
cd /var/www/harpyx
nano .env

Start from .env.example and replace every development value with production values.

Some settings are topology-dependent:

  • Database__Host and Database__Port should point to your real SQL Server
  • Minio__Endpoint should point either to the local Compose service or to the centralized object storage endpoint
  • RabbitMQ__HostName, Redis__ConnectionString, and OpenSearch__Endpoint should follow the same rule

Examples:

# all-container example
Database__Host=sqlserver
Database__Port=1433
Minio__Endpoint=minio:9000
RabbitMQ__HostName=rabbitmq
Redis__ConnectionString=redis:6379
OpenSearch__Endpoint=https://opensearch:9200
# centralized/shared services example
Database__Host=sql-prod.internal.example
Database__Port=1433
Minio__Endpoint=minio.internal.example:9000
RabbitMQ__HostName=rabbitmq.internal.example
Redis__ConnectionString=redis.internal.example:6379
OpenSearch__Endpoint=https://opensearch.internal.example:9200

Step 5 - Create the real production Compose file

Use docker-compose.prod.example.yml as the starting point for your real docker-compose.prod.yml.

If you want a fully containerized production host, you can keep most of the example as-is.

If you use centralized services, simplify it:

  • remove services you do not want this stack to own
  • remove matching depends_on entries where appropriate
  • keep only webapp, worker, and the backing containers that should actually run on this machine

For example, when SQL Server is centralized, the real production compose can omit the sqlserver service entirely and rely on .env for the external connection details.

Step 6 - First-time startup

If your chosen production topology includes local backing containers, start them before the first application deploy so they can become healthy:

cd /var/www/harpyx
docker compose -f docker-compose.prod.yml up -d
docker compose -f docker-compose.prod.yml logs -f

If some services are centralized, they obviously do not need to be started from this Compose project.

Step 7 - Nginx reverse proxy with TLS

Install Nginx and Certbot:

sudo apt install -y nginx certbot python3-certbot-nginx

Create /etc/nginx/sites-available/harpyx:

server {
    server_name yourdomain.com;

    location / {
        proxy_pass         http://localhost:8080;
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection keep-alive;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
        client_max_body_size 50m;
    }
}

Enable the site and request a TLS certificate:

sudo ln -s /etc/nginx/sites-available/harpyx /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
sudo certbot --nginx -d yourdomain.com

Step 8 - Choose a deployment mode

After the server is ready, you can choose between two valid deployment models.

Option 1 - GitHub Actions deploys over SSH

In this model, GitHub Actions not only builds and publishes the images, but also connects to the server and runs the deployment commands remotely.

Typical flow:

  1. Push to main or trigger the workflow manually
  2. GitHub Actions builds harpyx-webapp and harpyx-worker
  3. GitHub Actions pushes both images to ghcr.io
  4. GitHub Actions opens an SSH session to the server
  5. GitHub Actions runs the equivalent of:
cd /var/www/harpyx
docker compose -f docker-compose.prod.yml pull webapp worker
docker compose -f docker-compose.prod.yml up -d --no-deps --remove-orphans webapp worker
docker image prune -f

Required repository secrets for this mode:

  • SSH_HOST
  • SSH_USER
  • SSH_KEY
  • SSH_PASSPHRASE if your private key is encrypted
  • SSH_PORT
  • DEPLOY_PATH

This option is convenient when the server accepts inbound SSH from GitHub-hosted runners or from your chosen runner infrastructure.

Option 2 - No SSH from GitHub Actions

In this model, GitHub Actions only builds and publishes the images. Deployment is completed manually on the server.

Typical flow:

  1. Push to main or trigger the workflow manually
  2. GitHub Actions builds harpyx-webapp and harpyx-worker
  3. GitHub Actions pushes both images to ghcr.io
  4. You connect to the server yourself
  5. You run the deployment commands manually

Standard application deploy:

cd /var/www/harpyx
docker compose -f docker-compose.prod.yml pull webapp worker
docker compose -f docker-compose.prod.yml up -d --no-deps --remove-orphans webapp worker
docker image prune -f

First bootstrap or full stack refresh:

cd /var/www/harpyx
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d --remove-orphans
docker image prune -f

This option avoids exposing SSH access to GitHub Actions and is often simpler operationally. It may also be the only practical choice if the server is protected by a firewall allowlist and you are not using self-hosted runners or other runners with static IP addresses.

Subsequent deployments

Every push to main still triggers image build and publication automatically.

What happens next depends on the deployment mode you selected:

  • with SSH-based deployment, GitHub Actions can perform the rollout automatically
  • without SSH-based deployment, you run the docker compose commands manually on the server

Troubleshooting

Symptom Check
webapp or worker exits immediately Inspect docker compose -f docker-compose.prod.yml logs webapp and verify .env values, service endpoints, and credentials
docker pull fails with unauthorized The GHCR package is private: either make it public or authenticate the server to ghcr.io
Migrations fail on startup Verify Database__Host, Database__Port, credentials, reachability, and SQL Server TLS settings
OpenSearch health check never passes Verify password complexity and endpoint/TLS configuration
Wrong client IP in logs Review reverse-proxy forwarding and ReverseProxy__KnownNetworks
ClamAV takes several minutes on first boot Normal: signature database download can take time

Clone this wiki locally