Skip to content

Deployment

Jump to bottom
Valerio edited this page Apr 21, 2026 · 2 revisions

Deployment

Harpyx is packaged as two container images — one for the WebApp, one for the Worker — plus the six backing services needed at runtime. The reference deployment is docker-compose.yml; the same topology adapts directly to Kubernetes, Azure Container Apps, ECS, or a plain Docker host.

Topology

                      ┌─────────────────�
                      │  Reverse proxy  │
                      │ (Nginx / Caddy) │
                      └────────┬────────┘
                               │ :8080
                       ┌───────▼───────�
                       │  Harpyx.WebApp│───────�
                       └────────┬──────┘       │
                                │              │
      ┌─────────────────────────┼──────────────┼────────────────────────�
      │            │            │              │             │          │
      â–¼            â–¼            â–¼              â–¼             â–¼          â–¼
┌─────────�  ┌─────────�  ┌───────────�  ┌──────────�  ┌──────────� ┌─────────�
│  SQL    │  │  MinIO  │  │ RabbitMQ  │  │  Redis   │  │OpenSearch│ │ ClamAV  │
│ Server  │  │         │  │           │  │          │  │          │ │         │
└─────────┘  └─────────┘  └─────┬─────┘  └──────────┘  └──────────┘ └─────────┘
                                │
                        ┌───────▼────────�
                        │ Harpyx.Worker  │
                        └────────────────┘

Both application processes share the same SQL, MinIO, and OpenSearch backends. The WebApp writes metadata and blobs and publishes jobs; the Worker consumes jobs and writes chunks.

Docker Compose

The Compose file defines 8 services — 6 infrastructure + 2 application — and reads .env from the repo root for variable interpolation. See Configuration for the naming convention and the container-side __/-- separators.

The Compose project name is defined by:

COMPOSE_PROJECT_NAME=harpyx
DOTNET_ENVIRONMENT=Development
ASPNETCORE_URLS=http://+:8080

This keeps generated container, network, and volume names stable across local folders, and makes the shared runtime environment and WebApp container listener explicit.

Bringing it up

docker compose up --build -d
docker compose logs -f webapp worker

Health checks are declared on each infrastructure service, and the webapp / worker services use depends_on: condition: service_healthy to wait for their dependencies. First-time startup takes 1–3 minutes because:

  • MSSQL runs initial catalog setup.
  • OpenSearch initializes the security plugin and generates certificates.
  • ClamAV downloads its virus signature database before it can scan.

Container images

Both base on mcr.microsoft.com/dotnet/aspnet:10.0 and run as a non-root user.

Reverse proxy

In production, terminate TLS in front of the WebApp (Nginx, Caddy, Traefik, Azure Front Door, CloudFront, …). The WebApp trusts forwarded headers only from IP ranges listed under ReverseProxy:KnownProxies / ReverseProxy:KnownNetworks — set these to the private IPs of your proxy, not 0.0.0.0/0. ReverseProxy:ForwardLimit caps the number of forwarder hops honored.

CI/CD and Production Deployment

See CI/CD for the full guide covering the GitHub Actions pipeline, Ubuntu server setup, SSH key configuration, Nginx reverse proxy with TLS, and step-by-step first-time deployment instructions.

Scaling

Horizontal scale-out

Service Replicable? Notes
webapp Yes Behind a load balancer. Share DataProtection:KeysPath on a network volume (or switch to Key Vault / Redis DP key storage) so cookies are decryptable across instances.
worker Yes RabbitMQ distributes messages to consumers; the Job row + Redis idempotency prevent double processing.
SQL Server Vertical + AG Always On / Managed Instance for HA.
MinIO Yes Distributed mode (4+ nodes) for production.
RabbitMQ Cluster 3-node cluster + quorum queues.
Redis Cluster / Sentinel Not load-bearing; even a single node survives app restarts cleanly.
OpenSearch Cluster 3-node minimum, separate master-eligible nodes beyond that.
ClamAV Yes Scanner is stateless; scale horizontally if upload volume grows.

Vertical considerations

  • Worker is CPU- and memory-hungry during extraction/OCR; size pods/VMs accordingly.
  • OpenSearch embedding index size grows with corpus × EmbeddingDimensions; plan storage per-tenant.
  • SQL Server transactional volume scales with projects × documents × jobs; audit events are the fastest-growing table — archive policies recommended once tenants age.

Configuration in production

Switch secrets from .env to Azure Key Vault (or any other secret manager that matches the -- → : key-vault naming convention). See Configuration — Azure Key Vault for the wiring.

Minimum production-hardening checklist:

  • DOTNET_ENVIRONMENT=Production for both WebApp and Worker processes.
  • MalwareScan:Enabled=true, MalwareScan:FailClosed=true.
  • OpenSearch:AllowInsecureTls=false with valid certificates.
  • Database:ApplyMigrationsOnStartup=false; migrations applied out-of-band.
  • DataProtection:KeysPath on a persistent, shared volume if more than one WebApp instance.
  • ReverseProxy:KnownProxies populated with the real proxy IPs.
  • Encryption:MasterKey stored in Key Vault, rotated on a schedule, with re-encryption of LlmConnection API key values as part of rotation.

Observability in production

  • Export OpenTelemetry OTLP to a collector (OpenTelemetry:Otlp:Endpoint) that fans out to your metrics backend (Prometheus, Datadog, New Relic, …) and tracing backend (Jaeger, Tempo, …).
  • Ship Serilog through a structured sink (Seq, Loki, Elasticsearch) for searchable logs.
  • Alert on JobProcessedCounter{result=failed}, dead-letter queue depth, and ready health check flaps — see Observability.

Backup and recovery

Store What to back up Cadence suggestion
SQL Server Full + log backups Daily full, 15-min log (or managed instance PITR)
MinIO Bucket versioning + offsite replication Continuous
OpenSearch Snapshot repository to object storage Daily; regenerable from SQL + MinIO in a pinch
Redis None required Transient data only
RabbitMQ Topology (exchanges/queues) only Recreated on startup; messages are transient

SQL Server and MinIO are the system of record — a disaster recovery plan needs to guarantee both can be restored to a consistent point in time.

Clone this wiki locally