Skip to content

Deployment and Operations

Krishnendu Bhattacherjee edited this page Nov 10, 2026 · 2 revisions

Deployment and Operations

Local Compose stack

Docker Desktop with the Linux engine and Compose is required. The stack exposes only http://127.0.0.1:8080, has no authentication, and is intended for local use. It starts an empty PostgreSQL database; it does not import SQLite history. From the project root in PowerShell, create five distinct random passwords in an ignored runtime file, then start the services:

if (Test-Path .env.compose) { throw 'Keep the existing .env.compose; do not overwrite its passwords.' }
$names = 'TL_ADMIN_PASSWORD','TL_ANALYST_PASSWORD','TL_HUNT_PASSWORD','TL_EVALUATOR_PASSWORD','TL_TRIAGE_PASSWORD'
$bytes = [byte[]]::new(32)
$lines = foreach ($name in $names) { [System.Security.Cryptography.RandomNumberGenerator]::Fill($bytes); "$name=$([Convert]::ToHexString($bytes))" }
[System.IO.File]::WriteAllLines((Join-Path (Get-Location) '.env.compose'), $lines)
docker compose --env-file .env.compose up --build -d
docker compose --env-file .env.compose ps

Open the dashboard at http://127.0.0.1:8080; /health checks API database connectivity. PostgreSQL is not exposed on the host. The migration service creates separate logins; the API receives only analyst and read-only hunting credentials. Provider credentials can be supplied in .env.compose for explicitly initiated enrichment; do not put secrets in VITE_* variables or images. Compose environment variables are visible to users with Docker access. The memory limits total 2.625 GiB, excluding Docker's own overhead. docker compose --env-file .env.compose down stops the stack while retaining the database volume. Keep .env.compose: generating new passwords does not change the PostgreSQL administrator password in an existing volume.

PostgreSQL backup and restore

This procedure covers the local Compose PostgreSQL database, not SQLite files. It uses a data-only archive and the existing migrations to recreate the schema and grants. Store the archive and revision file together in an access-restricted, encrypted location outside the repository: they contain sensitive alerts, labels, reviews and other investigation data. Keep .env.compose securely and separately; the archive does not contain cluster roles, passwords or external provider keys. Choose a new backup directory that does not already contain these filenames.

Stop all writers, including the optional worker and any external CLI or evaluation processes, and do not run migrations during the backup. From the project root in PowerShell, with the database running, set $backupDir to that directory:

docker compose --env-file .env.compose --profile jobs stop api worker
$db = docker compose --env-file .env.compose ps -q db
if (-not $db) { throw 'Start the database first.' }
docker exec $db psql -X -U triagelens_admin -d triagelens -At -v ON_ERROR_STOP=1 -c 'SELECT version_num FROM triagelens.alembic_version' | Set-Content -Encoding utf8 (Join-Path $backupDir 'revision.txt')
if ($LASTEXITCODE -ne 0) { throw 'Could not read migration revision.' }
docker exec $db pg_dump -U triagelens_admin -d triagelens -Fc --data-only --exclude-table-data=triagelens.alembic_version -f /tmp/triagelens-backup.dump
if ($LASTEXITCODE -ne 0) { throw 'Backup failed.' }
docker cp "${db}:/tmp/triagelens-backup.dump" (Join-Path $backupDir 'triagelens.dump')
if ($LASTEXITCODE -ne 0) { throw 'Copy failed.' }
docker exec $db rm /tmp/triagelens-backup.dump

Resume the stack with docker compose --env-file .env.compose --profile jobs up -d if the worker is used; omit --profile jobs otherwise. Check the archive and revision file exist and regularly test recovery. A dump is a point-in-time snapshot, not continuous recovery; changes after it are not included.

Restore first into a new, isolated Compose project and empty volume, never over an existing database. Use the same trusted code and migrations as the backup; build the application image from that checkout before starting the target. The commands below use project name triagelens-recovery, which must not already exist. Use the protected credentials file intended for the target and ensure its optional queue password is set if the worker is needed. Stop and investigate any failed command. Do not restore an archive from an untrusted source.

docker compose --env-file .env.compose -p triagelens-recovery up -d --build db migrate
if ($LASTEXITCODE -ne 0) { throw 'Target startup failed.' }
$migration = docker compose --env-file .env.compose -p triagelens-recovery ps -aq migrate
if ((docker wait $migration) -ne '0') { throw 'Target migration failed.' }
$target = docker compose --env-file .env.compose -p triagelens-recovery ps -q db
$revision = docker exec $target psql -X -U triagelens_admin -d triagelens -At -v ON_ERROR_STOP=1 -c 'SELECT version_num FROM triagelens.alembic_version'
if ($LASTEXITCODE -ne 0 -or $revision.Trim() -ne (Get-Content (Join-Path $backupDir 'revision.txt') -Raw).Trim()) { throw 'Revision mismatch; do not restore.' }
docker cp (Join-Path $backupDir 'triagelens.dump') "${target}:/tmp/triagelens-restore.dump"
if ($LASTEXITCODE -ne 0) { throw 'Copy failed.' }
docker exec $target pg_restore -U triagelens_admin -d triagelens --exit-on-error --single-transaction /tmp/triagelens-restore.dump
if ($LASTEXITCODE -ne 0) { throw 'Restore failed; investigate before using target.' }
docker exec $target rm /tmp/triagelens-restore.dump

If restore fails, discard the isolated target and start again with an empty volume; sequence changes are not necessarily rolled back with the transaction. Before directing users or jobs to the recovered instance, verify representative records, independent labels, saved hunts, role permissions and application behavior. The migration recreates the version row and permissions; the archive deliberately excludes that row. The archive does not restore custom database settings, custom roles or changes outside the tracked migrations. In-flight jobs may be retried after recovery; check their state before enabling the worker. This procedure was tested with synthetic data on PostgreSQL 17, not as a production disaster drill.

SQLite remains the default. For a fresh UTF-8 PostgreSQL 17 database, install the optional dependencies and run the baseline as a migration owner with permission to create schemas and roles. Set TRIAGELENS_POSTGRES_MIGRATION_DSN in that process before running Alembic.

python -m pip install --only-binary=:all: -r requirements-postgres.txt
python -m alembic upgrade head

Provision separate non-owner logins for the four baseline roles and the optional worker role. Grant each login membership in just its corresponding group role. Supply distinct credentials through process-scoped environment variables.

Group role Connection variable Caller
triagelens_evaluator TRIAGELENS_POSTGRES_EVALUATOR_DSN CLI and evaluation coordinator
triagelens_triage TRIAGELENS_POSTGRES_TRIAGE_DSN Restricted inference cache
triagelens_analyst TRIAGELENS_POSTGRES_ANALYST_DSN API and job submission/status
triagelens_hunt TRIAGELENS_POSTGRES_HUNT_DSN Read-only hunting queries
triagelens_queue (optional) TRIAGELENS_POSTGRES_QUEUE_DSN Worker claims and job results

Set DATABASE_BACKEND=postgresql for the API, CLI, and standard evaluation runner. An explicit --db-path selects SQLite for that invocation. The dedicated live protocol requires this explicit SQLite path. The PostgreSQL baseline starts empty and leaves SQLite history in its original file.

Saved-hunt worker

The optional PostgreSQL worker runs an existing saved hunting filter and stores its returned row count, capped at the requested limit (1–1,000, default 500). Save the hunt in the dashboard first. Submission snapshots the filter. Use the same job ID, hunt name, and limit to retry a submission, including after the saved hunt is edited or removed. Use a fresh ID to run it again.

To enable the Compose worker, add a distinct TL_QUEUE_PASSWORD to the ignored .env.compose file. Generate it once, then keep it with the other credentials.

if (-not (Select-String -LiteralPath .env.compose -Pattern '^TL_QUEUE_PASSWORD=' -Quiet)) {
    $queueBytes = [byte[]]::new(32)
    [System.Security.Cryptography.RandomNumberGenerator]::Fill($queueBytes)
    Add-Content -LiteralPath .env.compose -Value "TL_QUEUE_PASSWORD=$([Convert]::ToHexString($queueBytes))"
}
docker compose --env-file .env.compose --profile jobs up --build -d
$jobId = [guid]::NewGuid().ToString()
docker compose --env-file .env.compose exec api python deploy/start.py queue enqueue 'My saved hunt' $jobId --limit 500
docker compose --env-file .env.compose exec api python deploy/start.py queue status $jobId

The worker adds a 256 MiB limit to the Compose stack. To stop the stack with the worker enabled, use docker compose --env-file .env.compose --profile jobs down.

For standalone PostgreSQL, apply python -m alembic upgrade head and grant triagelens_queue to a dedicated login. Set the queue and hunt DSNs in the worker process and run python -m src.worker run. In a separate process with the analyst DSN, use python -m src.worker enqueue 'My saved hunt' <job-id> and python -m src.worker status <job-id>. Use run --once for a single poll.

Claims use 30-second leases. Transient failures and expired leases can retry after 5 and 10 seconds, with at most three attempts. An expired attempt may repeat the read-only query against current data. The lease token prevents a stale attempt from publishing a result. Job rows and filter snapshots remain in PostgreSQL. Status shows a bounded count or a short failure code.

To run the offline regression checks, set TRIAGELENS_TEST_POSTGRES_DSN to a disposable PostgreSQL instance on 127.0.0.1 using a login that can create temporary test databases and roles. Then run:

python -m pytest -q -x --tb=short tests/test_database_contract.py tests/test_compose_startup.py

The trusted evaluator preserves independent labels and raw records. The triage login reads verdicts only. The hunt login queries a read-only analyst view that includes alert labels, with a server-side statement timeout. The raw record has a JSONB GIN index for supported containment queries. Time partitioning is disabled. Within each CLI or evaluation run, the trusted coordinator passes a label-filtered timeline and a restricted cache login to the inference stage in the same process.

For verification, point TRIAGELENS_TEST_POSTGRES_DSN at an administrative login on a disposable instance at 127.0.0.1. The tests create and remove a test database and test logins. The baseline group roles remain on the test instance.

python -m pytest -q -x --tb=short tests/test_database_contract.py

Wiki home · Project README

Clone this wiki locally