Distributed Deployment

Distributed Deployment

Use this topology when multiple hs-sql-agent replicas must share Admin/Auth data and enforce cluster-wide limits.

Required shared state

Concern	Provider
Admin/Auth database	PostgreSQL
Application cache	Redis when cache entries must be shared
IP and MCP-key quotas	Redis
Security-policy notification	Redis pub/sub plus PostgreSQL polling
SQL concurrency	Redis renewable leases

Do not mount one SQLite file into multiple replicas or place it on shared network storage.

Environment

Every replica must use the same PostgreSQL database, Redis endpoint, HMAC/JWT secrets, namespaces, and SQL concurrency key.

ADMIN_DATABASE_PROVIDER=Postgres
ADMIN_DATABASE_CONNECTION_STRING=Host=postgres;Port=5432;Database=hsqlagent;Username=hsqlagent;Password=<password>

CACHE_PROVIDER=Redis
CACHE_CONNECTION_STRING=redis:6379
CACHE_KEY_PREFIX=hsqlagent:cache:

RATE_LIMITER_PROVIDER=Redis
RATE_LIMITER_CONNECTION_STRING=redis:6379
RATE_LIMITER_FAILURE_MODE=FailClosed
RATE_LIMITER_KEY_PREFIX=hsqlagent:ratelimit:

SECURITY_POLICY_SYNC_PROVIDER=Redis
SECURITY_POLICY_SYNC_CONNECTION_STRING=redis:6379
SECURITY_POLICY_SYNC_KEY_PREFIX=hsqlagent:security-policy:
SECURITY_POLICY_SYNC_REFRESH_INTERVAL_SECONDS=30

SQL_CONCURRENCY_PROVIDER=Redis
SQL_CONCURRENCY_CONNECTION_STRING=redis:6379
SQL_CONCURRENCY_FAILURE_MODE=FailClosed
SQL_CONCURRENCY_KEY=hsqlagent:sql-concurrency
SQL_CONCURRENCY_LEASE_SECONDS=30

Cache, request limiting, policy sync, and SQL concurrency remain independent providers even when they share one Redis server.

Runtime behavior

• Request limits use atomic Redis counters shared by every replica.
• Policy changes publish immediately; PostgreSQL polling repairs missed notifications.
• SQL operations hold renewable Redis leases; expired leases recover capacity after a crash.
• MaxConcurrentSql is cluster-wide only when the Redis concurrency provider is selected.

The request limiter and SQL concurrency limiter support two Redis failure modes:

Mode	Behavior
FailClosed	Reject protected work while Redis is unavailable
FailOpen	Continue while temporarily bypassing the distributed limit

Use FailClosed unless availability requirements explicitly justify bypassing enforcement. Distributed request queuing is not implemented; keep RATE_LIMITING_QUEUE_LIMIT=0.

PostgreSQL migrations and data transfer

PostgreSQL uses provider-specific Admin and Auth migrations with separate history tables. Startup applies Auth migrations before Admin migrations.

Changing ADMIN_DATABASE_PROVIDER does not copy SQLite data. Before cutover:

1. Back up the SQLite file.
2. Export and import users, roles, permissions, MCP keys, database definitions, policy, and audit data.
3. Validate relationships, encrypted values, and key behavior against PostgreSQL.
4. Keep the old deployment offline but recoverable until verification completes.

Rolling deployment

1. Provision PostgreSQL and Redis with persistence and monitoring.
2. Deploy one replica and wait for migrations to finish.
3. Verify Admin login, MCP authentication, policy loading, and Redis connectivity.
4. Start the remaining replicas.
5. Send traffic only after every health check passes.

See Configuration for individual keys and Architecture for implementation details.