Skip to content

design assistant image blobs

sinrimin edited this page Oct 20, 2026 · 1 revision

Design: assistant transcript image blobs (#59)

Issue #59. Status: SHIPPED 2026-10-16 (migration v23; sqlite + memory drivers; settings imageRetentionDays). Companion to the #58 image policy in design-platform-assistant.md — that doc owns WHAT the model context does with images; this doc owns WHERE the bytes live.

1. Problem

#58 wired image attach end to end and persisted image blocks as inline base64 inside assistant_messages.blocks — every image lands in SQLite wholesale, and every listMessages load pulls all historical bytes into memory even when the model-context rebuild degrades them to placeholders. The transcript rows are the wrong place for megabyte payloads.

2. Shape

Persisted rows reference blobs; the bytes live outside the database.

browser (wire unchanged: base64 image blocks)
  → assistant:message.send
  → server INGEST: for each image block
        fileBlobs.put(bytes, mime, expiresAt = now + settings.imageRetentionDays)
        persisted block = { type:'imageRef', ref: <blob id>, mimeType }
  → rows hold ~100B refs, never bytes
  • No dedup: every ingest is its own object (one send, one blob). A blob is referenced by exactly one user row, which is what makes session-delete reclamation a straight collect-and-delete.
  • Storage follows the repo's existing pattern: a port in core (FileBlobRepository on the UnitOfWork), drivers in server/src/infra/storage/. The sqlite driver keeps a small registry table (file_objects: id / mime / size / expires_at / created_at) and writes each payload as one file under the data directory's files/ folder (the same volume the SQLite file lives on — container mounts keep them together); the DB row is inserted only after the payload file exists, so a reader can never observe a half-written object. The memory driver keeps both in a map (ephemeral by design).

Block arms (AssistantPromptBlock, core domain)

  • { type:'image', data, mimeType } — the LEGACY inline arm. The ingest path no longer produces it; pre-#59 rows stay readable everywhere (no migration: the blocks column is a JSON union and every reader handles both arms).
  • { type:'imageRef', ref, mimeType } — what the send path persists.

The WIRE schema (promptBlockSchema) is unchanged: the browser still sends base64; the split happens server-side at ingest.

3. Retention — stamped, never retroactive

SystemSettings.imageRetentionDays (default 30, integer 1–3650, admin-only GET/PUT /api/settings/image-retention; the admin settings page carries the field with the "applies to new images only" wording). The value becomes a deadline in exactly one place: imageExpiryFrom(settings, now) at INGEST. Each blob carries its own expiresAt from that moment and nothing ever rewrites it — changing the setting affects future ingests only.

Expiry is judged at READ time against the stamp (the sweep is reclamation, not a correctness gate):

  • Model context (loadImageMap → the #58 policy builders): an expired or missing ref degrades to the existing [image omitted] placeholder.
  • History replay / user_message echo / queue_state: refs resolve transiently to inline data URLs over the owner-authenticated socket (no new HTTP surface); an expired or missing ref renders as [image expired].

Reclamation:

  • Sweep: at boot and every 12h (app.ts, unref'd interval), deleting registry rows whose stamp has passed plus their payload files.
  • Session delete: deleteAssistantSession collects the rows' refs before the cascade and deletes those blobs (1:1 with rows, by the no-dedup rule).

4. Model-context details

loadImageMap preloads every ref the turn could need (prompt + rows) once; the pure builders (rowsToPiMessages / promptBlocksToPiContent) read the map, never the repository. One shape rule from the OpenAI-compatible wires: user content must not be images-only — a turn that would otherwise carry no text part gets a leading [image] text part (both arms, both builders).

5. Tests & rig verification

  • test/file-blobs.test.ts (5): sqlite round-trip (bytes on disk, never in the DB; payload-gone reads null; delete idempotent), purgeExpired selectivity, memory driver through the UnitOfWork, files-dir placement.
  • test/assistant-image-blobs.test.ts (4): send → row holds imageRef + blob exists + echo carries the inline wire view; retention stamped at ingest with the setting change NOT retroactive (first blob keeps its +1d stamp, second gets +30d); replay resolves fresh / degrades after purge; session delete reclaims the blob.
  • test/assistant-image-policy.test.ts (+5): imageRef arms through the builders, the [image] images-only guard, replay placeholder, and loadImageMap treating expired rows as missing.
  • Rig (2026-10-16, tui-verify + admin): migration v23 clean; live send → row ["text","imageRef"], /data/files/ payload present, stamp +30d; reload replays the image; setting 30→7 → new blob stamped +7d while the old kept +30d; hand-expired blob → [image expired] in the replay; server restart → boot sweep removed file+row; session delete emptied the store; setting restored to 30.

Clone this wiki locally