Skip to content

1.5 data

Jump to bottom
wiki[bot] edited this page Aug 2, 2026 · 3 revisions

1.5. Data & Storage

3F state lives in three stores, each chosen for its access shape:

Store Holds Accessed via
PostgreSQL 16 Conversations, dead-letter envelopes, per-session config Prisma 7
MinIO Raw image payloads (bytes) S3 API (modules/minio)
KeyDB Queue state, pub/sub, cancellation markers, ephemeral caches BullMQ / ioredis

Prisma schema (server/prisma/schema.prisma)

The generated client lives at server/src/generated/prisma (checked-in build artifact of prisma generate, produced during image builds and the install step). Prisma 7's prisma-client generator is used; the datasource URL comes from environment (POSTGRES_URL via prisma.config.ts).

HarnessConversation → harness_conversation

One row per exchange request: composite key (sessionId, conversationId, requestId), optional title, and JSON content holding the full exchange (prompt, messages, phases, structured results). Indexed by (sessionId, conversationId) for session-scoped loading — this is what lets the dashboard reopen a conversation exactly where it left off, compacted history included.

HarnessDlq → harness_dlq

Persisted dead-letter envelopes keyed by requestId:

  • identity: queueName, jobId;
  • status lifecycle: DlqStatus — Failed → Active → Cleared → Removed;
  • context: payload (JSON), retryConfig, failureHistory, failedReason;
  • bookkeeping: attemptsMade / totalAttempts, failedAt, nextRetryAt.

See 1.3 for how records arrive here and 1.1 for the DLQ API that manages them.

HarnessConfig → harness_config

Per-session workbench configuration (key sessionId): selectedModel, preprocessing (JSON), and providerOverrides (JSON). This is the server-side state behind the SysCtl area (see 2-dashboard): overrides set in the UI are persisted here and applied by the harness on the next request.

MinIO — image payloads

Images uploaded with POST /harness are stored per conversation, addressed by content hash:

storage/:sessionId/:conversationId/:hash

Properties:

  • Dedup by construction — the dashboard sends known hashes in sessionMetadata; the storage API exposes an exists probe, so identical images are never re-uploaded or re-stored.
  • Stable addressing — hash-as-key makes payloads idempotent and replay-safe (DLQ reinstatement can re-reference the original bytes).
  • Lifecycle — whole-conversation deletion is one call (DELETE /storage/:sessionId/:conversationId).

MinioHealthIndicator participates in /health/ready; bucket bootstrap happens at startup.

KeyDB

Configuration (server/keydb.conf, mounted into the container): password-protected (requirepass redis for dev), 500 MB memory ceiling, 2 I/O threads, RDB persistence. It backs:

  1. BullMQ structures (waiting/active/failed sets, delayed jobs);
  2. Socket.IO adapter state for multi-instance room routing;
  3. Lightweight runtime markers (cancellation, overrides cache).

Everything on KeyDB is rebuildable — the durable truth is PostgreSQL + MinIO. Wiping the keydb_data volume mid-session costs, at most, in-flight queued jobs (and those get re-instated, see 1.3).

Image preprocessing (sharp)

modules/sharp generates image variants before model consumption — resized/normalized derivatives that improve vision-model throughput and OCR quality without touching the stored original. Effective settings are configurable per session (HarnessConfig.preprocessing), pushed from the dashboard's PProc/SysCtl controls; the server treats client overrides as the effective config on the next request.

Clone this wiki locally