Skip to content

1.5 data

Jump to bottom
wiki[bot] edited this page Aug 8, 2026 · 3 revisions

1.5. Data & Storage

3F state lives in three stores, each chosen for its access shape:

Store Holds Accessed via
PostgreSQL 16 Conversations, dead-letter envelopes, per-session config Prisma 7
MinIO Raw image payloads (bytes) S3 API (modules/minio)
KeyDB Queue state, pub/sub, cancellation markers, ephemeral caches BullMQ / ioredis

Prisma schema (server/prisma/schema.prisma)

The generated client lives at server/src/generated/prisma (checked-in build artifact of prisma generate, produced during image builds and the install step). Prisma 7's prisma-client generator is used; the datasource URL comes from environment (POSTGRES_URL via prisma.config.ts).

HarnessConversation → harness_conversation

One row per exchange request: composite key (sessionId, conversationId, requestId), optional title, and JSON content holding the full exchange (prompt, messages, phases, structured results). Indexed by (sessionId, conversationId) for session-scoped loading — this is what lets the dashboard reopen a conversation exactly where it left off.

HarnessDlq → harness_dlq

Persisted dead-letter envelopes keyed by requestId:

  • identity: queueName, jobId;
  • status lifecycle: DlqStatus — Failed → Active → Cleared → Removed;
  • context: payload (JSON), retryConfig, failureHistory, failedReason;
  • bookkeeping: attemptsMade / totalAttempts, failedAt, nextRetryAt.

See 1.3 for how records arrive here and 1.1 for the DLQ API that manages them.

HarnessConfig → harness_config

Per-session workbench configuration (key sessionId): selectedModel, preprocessing (JSON), and providerOverrides (JSON). This is the server-side state behind the SysCtl area (see 2-dashboard): overrides set in the UI are persisted here and applied by the harness on the next request.

HarnessShownMedia → harness_shown_media

Media the user has already been shown inside a conversation, keyed by deterministic identity: image entries carry a normalized content fingerprint (fp:) or storage hash (sh:), video entries carry canonical provider keys. Written at respond time from the guarded final data; read at sanitize time so media-list follow-ups never repeat already-shown content. Rows are purged when their conversation or session is deleted.

HarnessProviderOverride → harness_provider_override

Per-provider override rows (key provider): serper, brightData, youtube, sources. values holds the provider config (API keys encrypted at rest — see the secrets cipher below).

HarnessPlaylist → harness_playlist

A named playlist scoped to one conversation (composite key (sessionId, conversationId, name)). videos is a JSON array of VideoGalleryItem; the active playlist is the queue. Rows are purged when their conversation is deleted. Backed by the /api/v1/playlists controller (see 1.1).

MinIO — image payloads

Images uploaded with POST /harness are stored per conversation, addressed by content hash:

storage/:sessionId/:conversationId/:hash

Properties:

  • Dedup by construction — the dashboard sends known hashes in sessionMetadata; the storage API exposes an exists probe, so identical images are never re-uploaded or re-stored.
  • Stable addressing — hash-as-key makes payloads idempotent and replay-safe (DLQ reinstatement can re-reference the original bytes).
  • Lifecycle — whole-conversation deletion is one call (DELETE /storage/:sessionId/:conversationId).

MinioHealthIndicator participates in /health/ready; bucket bootstrap happens at startup.

KeyDB

Configuration (server/keydb.conf, mounted into the container): password-protected (requirepass redis for dev), 500 MB memory ceiling, 2 I/O threads, RDB persistence. It backs:

  1. BullMQ structures (waiting/active/failed sets, delayed jobs);
  2. Socket.IO adapter state for multi-instance room routing;
  3. Lightweight runtime markers (cancellation, overrides cache).

Everything on KeyDB is rebuildable — the durable truth is PostgreSQL + MinIO. Wiping the keydb_data volume mid-session costs, at most, in-flight queued jobs (and those get re-instated, see 1.3).

Secrets cipher

Provider API keys stored in HarnessProviderOverride are encrypted at rest with AES-256-GCM (modules/secrets, SecretsCipherService, keyed by TRIPLEF_SECRETS_KEY). The provider-overrides API returns masked keys and decrypts only when a provider actually needs the value, so secrets never appear in plaintext in the database or the UI.

Image preprocessing (sharp)

modules/sharp generates image variants before model consumption — resized/normalized derivatives that improve vision-model throughput and OCR quality without touching the stored original. Effective settings are configurable per session (HarnessConfig.preprocessing), pushed from the dashboard's PProc/SysCtl controls; the server treats client overrides as the effective config on the next request.

Clone this wiki locally