Skip to content

sessions: replace per-step LLM request bodies with hash-deduped prompt envelopes - #484

Merged
ReganBell merged 1 commit into
mainfrom
upstream-1781
Aug 13, 2026
Merged

sessions: replace per-step LLM request bodies with hash-deduped prompt envelopes#484
ReganBell merged 1 commit into
mainfrom
upstream-1781

Conversation

@ReganBell

@ReganBell ReganBell commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

The per-step LLM request log stored the full provider payload on every model-call step — the entire growing conversation re-serialized per step, O(turns squared) bytes — and it dominates database growth on any active deployment; the message arrays it duplicates are already durable exactly once on the session tape. Replace the stored request body with a compact prompt envelope: system/tools/config captured once and deduplicated by content hash, with the message window reconstructed from the tape on read. Observability views reassemble the original request for inspection, so debugging fidelity is unchanged while storage per step drops from the full conversation to a constant-size record.

Deployment notes

  • Release note: admin LLM forensics for steps recorded by the new version render without request bodies on any pre-upgrade instance or after rollback; scrub/snapshot pipelines must add llm_prompt_envelopes to exclusions.
  • Schema at boot, all idempotent and instant: ALTER session_llm_requests ALTER COLUMN request DROP NOT NULL, ADD COLUMN IF NOT EXISTS prompt_hash, CREATE TABLE IF NOT EXISTS llm_prompt_envelopes. No table rewrite; pre-migration request bodies are frozen in place and detail reads fall back to them — old rows keep serving.
  • Rollback / blue-green hazard: new code writes request = NULL + prompt_hash. Old code's rowToLlmRequest and the admin viewer expect request NOT NULL semantics — during a blue-green overlap or after rollback, old instances read NULL-request rows and render blank context for those steps. The ALTER already dropped the constraint, so writes do not fail; this is a read-quality gap, forensics-only, not user-facing.
  • No backfill is needed or attempted by design; the change is safe for small databases.
  • Follow-up: snapshot/scrub pipelines that exclude session_llm_requests must also exclude llm_prompt_envelopes.
  • Known follow-up: envelope blobs are not garbage-collected when sessions are deleted — slow, mild, unbounded growth.

Screenshots of the admin viewer were omitted from this PR; the viewer change reassembles the same request view as before, with a blank-context fallback for NULL-request rows.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…t envelopes

The per-step LLM request log stored the full provider payload on every model-call step — the entire growing conversation re-serialized per step, O(turns squared) bytes — and it dominates database growth on any active deployment; the message arrays it duplicates are already durable exactly once on the session tape. Replace the stored request body with a compact prompt envelope: system/tools/config captured once and deduplicated by content hash, with the message window reconstructed from the tape on read. Observability views reassemble the original request for inspection, so debugging fidelity is unchanged while storage per step drops from the full conversation to a constant-size record.
@ReganBell
ReganBell merged commit 73d793b into main Aug 13, 2026
21 checks passed
@ReganBell
ReganBell deleted the upstream-1781 branch August 13, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant