# 1.5. Data & Storage

3F state lives in four stores, each chosen for its access shape:

| Store         | Holds                                                        | Accessed via             |
| ------------- | ------------------------------------------------------------ | ------------------------ |
| PostgreSQL 16 | Conversations, dead-letter envelopes, per-session config     | Prisma 7                 |
| MinIO         | Raw image payloads (bytes)                                   | S3 API (`modules/minio`) |
| KeyDB         | Queue state, pub/sub, cancellation markers, ephemeral caches | BullMQ / ioredis         |
| Qdrant        | Long-term memory records + the AI's cognition document       | REST (`modules/qdrant`)  |

## Prisma schema (`server/prisma/schema.prisma`)

The generated client lives at `server/src/generated/prisma` (checked-in build artifact of `prisma generate`, produced during image builds and the install step). Prisma 7's `prisma-client` generator is used; the datasource URL comes from environment (`POSTGRES_URL` via `prisma.config.ts`).

### `HarnessConversation` → `harness_conversation`

One row per **exchange request**: composite key `(sessionId, conversationId, requestId)`, optional `title`, and JSON `content` holding the full exchange (prompt, messages, phases, structured results). Indexed by `(sessionId, conversationId)` for session-scoped loading — this is what lets the dashboard reopen a conversation exactly where it left off.

### `HarnessDlq` → `harness_dlq`

Persisted dead-letter envelopes keyed by `requestId`:

- identity: `queueName`, `jobId`;
- status lifecycle: `DlqStatus` — `Failed` → `Active` → `Cleared` → `Removed`;
- context: `payload` (JSON), `retryConfig`, `failureHistory`, `failedReason`;
- bookkeeping: `attemptsMade` / `totalAttempts`, `failedAt`, `nextRetryAt`.

See **1.3** for how records arrive here and **1.1** for the DLQ API that manages them.

### `HarnessConfig` → `harness_config`

Per-session workbench configuration (key `sessionId`): `selectedModel`, `preprocessing` (JSON), `providerOverrides` (JSON), and the memory space ids `memoryPartition` / `memoryCognition` — the user-set memory overrides (see the memory section below). This is the server-side state behind the SysCtl area (see **2-dashboard**): overrides set in the UI are persisted here and applied by the harness on the next request.

### `HarnessShownMedia` → `harness_shown_media`

Media the user has already been shown inside a conversation, keyed by deterministic identity: image entries carry a normalized content fingerprint (`fp:`) or storage hash (`sh:`), video entries carry canonical provider keys. Written at respond time from the guarded final data; read at sanitize time so media-list follow-ups never repeat already-shown content. Rows are purged when their conversation or session is deleted.

### `HarnessProviderOverride` → `harness_provider_override`

Per-provider override rows (key `provider`): `serper`, `brightData`, `youtube`, `sources`, `ollama`, `memory`. `values` holds the provider config (API keys encrypted at rest — see the secrets cipher below). The `memory` row carries the memory system variables (`cognitionLimit` — the cognition profile character cap), written by `PUT /memory-overrides` and layered over the `MEMORY_COGNITION_LIMIT` env baseline.

### `HarnessPlaylist` → `harness_playlist`

A named playlist scoped to one conversation (composite key `(sessionId, conversationId, name)`). `videos` is a JSON array of `VideoGalleryItem`; the active playlist is the queue. Rows are purged when their conversation is deleted. Backed by the `/api/v1/playlists` controller (see **1.1**).

### `StockMarketBar` → `stock_market_bar`

Cached end-of-day OHLCV bar for one ticker and trading date (composite key `(ticker, date)`): `open`/`high`/`low`/`close`, optional `adjustedClose`, `volume`, `fetchedAt`. Final bars are immutable; the repository re-fetches the most recent days so late updates (post-close uploads, restated adjusted closes) propagate.

### `StockMarketHistoryRange` → `stock_market_history_range`

Ledger of the `[from, to)` date windows already fetched per ticker (composite key `(ticker, fromDate, toDate)`). Trading days skip weekends/holidays, so coverage cannot be derived from the bars alone — requests only backfill intervals not in this ledger. Together the two tables back the `/api/v1/stock-data` endpoints (see **1.1**).

## MinIO — image payloads

Images uploaded with `POST /harness` are stored per conversation, addressed by **content hash**:

```
storage/:sessionId/:conversationId/:hash
```

Properties:

- **Dedup by construction** — the dashboard sends known hashes in `sessionMetadata`; the storage API exposes an `exists` probe, so identical images are never re-uploaded or re-stored.
- **Stable addressing** — hash-as-key makes payloads idempotent and replay-safe (DLQ reinstatement can re-reference the original bytes).
- **Lifecycle** — whole-conversation deletion is one call (`DELETE /storage/:sessionId/:conversationId`).

`MinioHealthIndicator` participates in `/health/ready`; bucket bootstrap happens at startup.

## KeyDB

Configuration (`server/keydb.conf`, mounted into the container): password-protected (`requirepass redis` for dev), 500 MB memory ceiling, 2 I/O threads, RDB persistence. It backs:

1. BullMQ structures (waiting/active/failed sets, delayed jobs);
2. Socket.IO adapter state for multi-instance room routing;
3. Lightweight runtime markers (cancellation, overrides cache).

Everything on KeyDB is **rebuildable** — the durable truth is PostgreSQL + MinIO. Wiping the `keydb_data` volume mid-session costs, at most, in-flight queued jobs (and those get re-instated, see **1.3**).

## Qdrant — long-term memory

Feature-gated (`MEMORY_ENABLED`, default off). One collection per embedding model (name suffixed with the model; vector size probed from the live model at bootstrap, Cosine distance). Payload keyword indexes cover every filtered field; the two space identity keys carry Qdrant's `is_tenant` multitenancy marker.

**One point = one record whose payload text IS the record** (no chunk layer — the transcript already lives in `HarnessConversation`). Point ids are deterministic (`sha256` of a seed), so re-stating a record overwrites it in place. Every point belongs to exactly one space, folded into the identity key:

- `memory_partition` — the **user's fact space**: statements the user made or asked to remember (preferences, contact details, notable gathered facts). Written two ways: the agentic `memoryRemember` tool (sync store) and the background vectorize pipeline (`extract → embed → store`, one job per turn-side, roles `user`/`assistant`).
- `memory_cognition` — the **AI's cognition space**: the assistant's derived, accumulated understanding of the user in two forms. (1) ONE structured **profile** document per cognition key (tags `['cognition','profile']`): stable identity and durable traits — `name`, `language`, `timezone`, `expertise[]`, `goals[]`, `communication{style,detailLevel,formality}`, `preferences{}`, `likes[]`, `dislikes[]`, `interests[]` — rewritten wholesale by the `memory-profile` vectorize-queue job after every answered turn, serialized JSON capped by the `memoryCognitionLimit` system variable (env baseline `MEMORY_COGNITION_LIMIT`, default 5000, clamped 500–32000; runtime override via `PUT /memory-overrides`, persisted globally in the `provider_overrides` row `memory`). (2) Derived **insight** records (tags `['cognition','insight']`, ≤500 chars each, id seeded on the text so repeats overwrite): the DEPTH behind profile topics — the profile doubles as the routing map: its lean topic values ("cars", "Linux") token-match against the prompt at respond time (word-boundary, deterministic — no extra model call) and sharpen the insight query for the hot facet (`likes.cars` style paths are stored on each insight for observability); probe fires only when insights exist. Both personalize every answer as marked private system context (never quoted as user-stated facts; disclosed plainly when asked); the user can also wipe the space (`memoryDelete` with `cognition=true` — purges profile + insights).

**Reads**: `MemorySearchService` embeds the query as the full text AND per-sentence variants, merges hits by best score — sentence-dense records are matched on their strongest segment. Records have no exposed ids: exact-text equality is the deletion identity.

**Retention/deletion**: sysctl prune (`DELETE /qdrant/memory`) wipes a partition (facts AND cognition) or one conversation (facts only — cognition transcends conversations by design); `DELETE /qdrant/text` deletes by filters (exact text / contains / tags / conversation / request, capped at 50 matches per call) or the cognition space (`cognition=true` — batched purge of profile + insights). Unscoped deletes are rejected.

## Secrets cipher

Provider API keys stored in `HarnessProviderOverride` are encrypted at rest with **AES-256-GCM** (`modules/secrets`, `SecretsCipherService`, keyed by `TRIPLEF_SECRETS_KEY`). The provider-overrides API returns masked keys and decrypts only when a provider actually needs the value, so secrets never appear in plaintext in the database or the UI.

## Image preprocessing (sharp)

`modules/sharp` generates image variants before model consumption — resized/normalized derivatives that improve vision-model throughput and OCR quality without touching the stored original. Effective settings are configurable per session (`HarnessConfig.preprocessing`), pushed from the dashboard's PProc/SysCtl controls; the server treats client overrides as the effective config on the next request.