Repository navigation
Scaling Horizontally
Claude edited this page Sep 14, 2026
·
2 revisions
A single api + single worker on one well-specced VM is plenty for one
operator or a handful of orgs, and is the default. When you need more, both
tiers scale horizontally without a message broker — just Postgres:
-
Scaling the
worker. Background work runs off a Postgres work queue (background_jobs): the leader enqueues due jobs, and any number of workers claim them withSELECT … FOR UPDATE SKIP LOCKED, so they never double-process. Run more withdocker compose up -d --scale worker=3— heavy report ingestion then parallelizes across replicas. Exactly one replica auto-elects itself leader (a Postgres advisory lock) to run the schedule; if it dies another takes over automatically. Each worker also serves a liveness endpoint (:8080/health) that reports unhealthy if its loop stalls, so an orchestrator recycles a wedged — not just crashed — replica. -
Scaling the
api. It's stateless (sessions live in Postgres), so it scales out freely — with one caveat: the auth rate limiter defaults to in-memory per-process. Running multipleapireplicas? SetRATE_LIMIT_BACKEND=postgresso the limit is shared and correct across them; it's dependency-free (reuses Postgres) and the auth-endpoint volume is tiny. - No Redis, no Celery — on purpose. Reusing the datastore you already run and have hardened (RLS, encryption-at-rest, backups) keeps the attack surface and the ops burden down versus adding a broker.
-
Connection pooling (PgBouncer). Point
DATABASE_URLat a PgBouncer instance once replica count makes raw connection count matter. SetLEADER_DATABASE_URLto a direct, unpooled connection, since the leader's advisory lock is session-scoped and would misbehave under transaction pooling. -
Read replicas for report/analytics queries. Set
DATABASE_READ_URLto offload report/trend/posture dashboard endpoints there — unset by default, entirely opt-in. Known limitation: three of these endpoints can write to a small sender-identity cache on a miss, so don't point this at a read-only replica yet — what IS safe today is a second connection to the same primary. -
Still single by default: Postgres and the resolver. Use a managed,
backed-up Postgres instance in production, or add a read replica per
above. Postgres sharding is a deferred future option — the
RLS-by-
organization_iddata model already gives it a natural shard key if a single primary's vertical ceiling ever becomes the bottleneck.
Full env var reference → Configuration Reference.
← Back to the README · Live demo · Report an issue · MIT licensed
- Quick Start: Docker Compose
- Deploying with Portainer
- Deploying on Azure
- Configuration Reference
- Onboarding an Organization
- IMAP Mailbox
- Google Workspace Mailbox
- Sign-in Providers
- Multi-Tenancy & RLS
- DMARC Subdomain Inheritance
- Report Attribution
- SPF Conditional Mode
- Detected Domains
- DNS Check Scheduling
- Domain Rating
🏗️ Architecture
🗺️ Roadmap