From 6f8ab5b41e498ba5820c38cb46cacf71d40b82d1 Mon Sep 17 00:00:00 2001 From: Etai Lev Ran Date: Wed, 8 Jul 2026 21:06:19 +0300 Subject: [PATCH 1/2] docs: remove stale files and apply inline feedback --- docs/adr/0001-go-language.md | 29 -- docs/adr/0002-proxy-charon-two-phase-split.md | 47 --- docs/adr/0003-checkpoint-strategy.md | 35 -- docs/adr/0004-kv-over-sql.md | 44 -- docs/architecture.md | 139 +++---- docs/implementation-plan.md | 388 ------------------ docs/storage.md | 102 ++--- docs/terminology.md | 67 --- 8 files changed, 83 insertions(+), 768 deletions(-) delete mode 100644 docs/adr/0001-go-language.md delete mode 100644 docs/adr/0002-proxy-charon-two-phase-split.md delete mode 100644 docs/adr/0003-checkpoint-strategy.md delete mode 100644 docs/adr/0004-kv-over-sql.md delete mode 100644 docs/implementation-plan.md delete mode 100644 docs/terminology.md diff --git a/docs/adr/0001-go-language.md b/docs/adr/0001-go-language.md deleted file mode 100644 index d3afa94..0000000 --- a/docs/adr/0001-go-language.md +++ /dev/null @@ -1,29 +0,0 @@ -# ADR 0001: Go as Implementation Language - -**Status:** Accepted - -## Context - -The workload is I/O-bound: embedded KV writes (Pebble), HTTP handling, zstd compression. The primary deployment target is on-premises infrastructure for AI teams running self-hosted inference backends. Charon is an internal service (separate repo from the client-facing proxy). - -The core alternatives were Go and Rust. - -## Decision - -Go. - -## Reasons - -**No measurable performance gap for this workload.** Rust's advantages — zero-cost abstractions, deterministic allocation, fine-grained memory control — pay dividends on CPU-bound code. For I/O-bound work that blocks on disk or network syscalls, Go and Rust produce programs with effectively identical runtime characteristics. - -**Pure-Go storage stack with no CGo.** The storage layer uses [Pebble](https://github.com/cockroachdb/pebble), a pure-Go embedded KV engine. It cross-compiles to any target with `GOOS`/`GOARCH` and produces a fully static binary with `CGO_ENABLED=0`. The Rust equivalent would require C toolchain dependencies for embedded storage. This friction simply does not exist in Go and is the single most important practical difference for the single-binary deployment goal. - -**Goroutines fit the concurrency model naturally.** Background staging reaper, TTL expiry, and per-request chain-walk goroutines are all naturally expressed as goroutines communicating over channels with `context.Context` for cancellation. The Rust equivalent — `tokio` tasks, `Arc>`, `await` across lock boundaries — is correct but adds ceremony to sequential background bookkeeping. - -**The repository is already Go.** Minimizes contributor ramp-up. - -## Trade-offs Accepted - -Rust's `serde` + enum variants would be cleaner for the discriminated union of output item types (`message`, `function_call`, `function_call_output`, `reasoning`, `compaction`). In Go, this requires an interface with type-switch or a tagged struct with `json.RawMessage` plus a second unmarshal pass. - -Mitigation: generate Go types from the OpenAPI spec rather than hand-writing them. diff --git a/docs/adr/0002-proxy-charon-two-phase-split.md b/docs/adr/0002-proxy-charon-two-phase-split.md deleted file mode 100644 index a7396b0..0000000 --- a/docs/adr/0002-proxy-charon-two-phase-split.md +++ /dev/null @@ -1,47 +0,0 @@ -# ADR 0002: Proxy/Charon Two-Phase Split - -**Status:** Accepted - -## Context - -The OpenAI Responses API must accept stateful requests (with `previous_response_id`) and forward a flat context to a stateless inference backend. The question is how to divide responsibilities between the client-facing component (the proxy) and the storage/resolution service (Charon). - -Alternative architectures considered: - -- **Charon as the client-facing API**: Charon owns SSE, WebSocket, auth, TLS, and model routing. The proxy is eliminated or reduced to a simple load balancer. -- **Single-component design**: One process handles everything — client API, chain resolution, inference forwarding, and storage. -- **Charon as a library**: Embedded into the proxy; no network boundary between them. - -## Decision - -Charon is an internal storage/resolution service (separate repository from the proxy). The proxy owns all client-facing concerns. They interact via a staging protocol over an internal HTTP API: - -**Continuation turns** (has `previous_response_id`): -1. **Open staging** — `POST /staging?prev={previous_response_id}`: proxy receives `{staging_id, flat_context[]}`. Charon assembles history and creates a staging record. -2. **Inference** — proxy forwards to the inference server, which assigns the canonical response ID in its first streaming chunk. -3. **Append chunks** — proxy delivers response bytes incrementally via `PUT /staging/{staging_id}/chunks/{k}`. -4. **Complete** — `PUT /staging/{staging_id}/complete?response_id={canonical_id}&total={N}`: Charon commits the node atomically. - -**New chains** (no `previous_response_id`): -- No staging call needed. Inference server assigns the canonical response ID. -- After inference: buffered `POST /responses` (or staging protocol as above; skipped if `store: false`). - -## Reasons - -**Separation of scaling axes.** Proxy instances are stateless and scale horizontally without coordination. Charon scales independently based on storage throughput. Coupling them forces both to scale together. - -**`store: false` is a proxy concern, not a storage concern.** Connection-local ephemeral caches for WebSocket sessions belong in the component that owns the connection. Charon should not need to understand connection lifecycle. - -**Model routing is an infrastructure concern.** Charon can be deployed per inference model (single-binary mode) or as a shared service fronting multiple backends. Neither deployment shape requires Charon to be the client-facing component. - -**Auth, TLS, and streaming belong in the proxy.** These are operational concerns that change independently of storage logic. Keeping them in the proxy allows Charon's API to be simple and internal. - -## Trade-offs Accepted - -**Two network hops per request** (proxy → Charon resolve, proxy → inference, proxy → Charon store). In the single-binary deployment, these are in-process calls with no real network cost. In multi-instance deployment, the resolve call adds latency proportional to chain-walk time, not raw network RTT. - -**Canonical IDs owned by the inference server.** The inference server (vLLM, etc.) assigns the canonical response ID returned in its first streaming chunk. The proxy uses this ID in `response.created` to the client and in the store call to Charon. Charon mints only a short-lived `staging_id` at resolve time for ingest correlation — this is never client-visible. When the inference backend returns a non-`resp_` format ID (e.g. `chatcmpl-`), the proxy translates it before forwarding. - -**No staging cleanup needed for `store: false`.** Because staging records are created only at `POST /staging` time (for continuations), a `staging_id` minted at resolve that the proxy never commits leaves no orphaned chain node — the staging reaper cleans up the record TTL. - -**Chunked streaming store is implemented.** The proxy delivers output to Charon in chunks via the staging protocol (`PUT /staging/{id}/chunks/{k}` + `PUT /staging/{id}/complete`). See [Streaming Ingest](../architecture.md#streaming-ingest). diff --git a/docs/adr/0003-checkpoint-strategy.md b/docs/adr/0003-checkpoint-strategy.md deleted file mode 100644 index 02e6a48..0000000 --- a/docs/adr/0003-checkpoint-strategy.md +++ /dev/null @@ -1,35 +0,0 @@ -# ADR 0003: Checkpoint Strategy for Chain Reconstruction - -**Status:** Accepted - -## Context - -Responses form a singly-linked list. To reconstruct the flat context for a new turn, Charon must walk the chain from the current head back to the root and assemble all input/output items in order. Three strategies are viable: - -| Strategy | Write cost | Read cost | -|----------|-----------|-----------| -| Delta-per-response | O(1) | O(N) — full chain walk | -| Full-snapshot-per-response | O(N) | O(1) — single fetch | -| Checkpoint every K turns | O(1) amortized | O(K) — walk at most K steps | - -## Decision - -Checkpoint every K turns (configurable, default 10). - -Every K-th response stores a full materialized snapshot of the flat context at that position. A resolve call walks backward from the chain head until it hits a checkpoint, then reads forward from the checkpoint. Worst-case walk is K steps. - -## Reasons - -**Delta-per-response is unbounded.** A 500-turn conversation requires reading 500 payload files and making 500 IndexStore lookups on every new request. Latency grows linearly with conversation length with no bound. - -**Full-snapshot-per-response has unacceptable write amplification.** Turn N requires writing the entire prior context (N payloads). A 500-turn conversation writes a 500-item payload on turn 500. Total storage grows as O(N²). - -**Checkpoints bound both costs.** Write amplification is O(K) at most (the checkpoint payload grows with conversation length, but only written every K turns, not every turn). Read cost is O(K) regardless of total chain depth. K=10 keeps reads fast without significant write overhead. - -**Checkpoints are co-located with chain payloads.** Using `chain_root_id + position` in the filesystem path means checkpoint files are in the same directory as their chain's individual payloads. The full chain is enumerable by directory listing independent of the database. - -## Trade-offs Accepted - -**Checkpoint writes are larger than delta writes.** A checkpoint at position 100 in a long conversation includes all 100 prior turns' items. This is the cost of bounding read latency. - -**K is a tunable constant, not adaptive.** A smarter strategy would checkpoint based on item count (tokens) rather than turn count, since turn length varies widely. K-by-turn is simpler to implement and sufficient for Phase 1. Adaptive checkpointing can be added later without changing the storage format. diff --git a/docs/adr/0004-kv-over-sql.md b/docs/adr/0004-kv-over-sql.md deleted file mode 100644 index 5ca83fb..0000000 --- a/docs/adr/0004-kv-over-sql.md +++ /dev/null @@ -1,44 +0,0 @@ -# ADR 0004: Embedded Key-Value Store (Pebble) over SQL - -**Status:** Accepted - -## Context - -Charon must durably store conversation turns (request and response blobs keyed by response ID) and support two query patterns: - -1. **Point lookup by response ID** — `GET /responses/{id}`, parent-pointer fetch during chain walk. -2. **Sequential parent-pointer walk** — resolve `previous_response_id` chains from leaf to root by following a linked list of keys. - -There are no joins, no predicate-based queries over payload content, and no schema evolution requirements driven by query shape. Blobs are opaque to the storage layer; only the key (`NodeID`) and a small set of chain-linkage metadata fields are structured. - -The primary alternatives considered were: - -- **Relational database (SQLite or embedded PostgreSQL via pgx/pgembedded)**: Rich query language, mature tooling, ACID transactions. -- **Embedded key-value store (Pebble)**: Point lookups and sequential scans only, no query language, no external process. -- **Object store (S3-compatible or local filesystem blobs + SQLite index)**: Decouples metadata from blobs; adds operational complexity. - -## Decision - -Use [Pebble](https://github.com/cockroachdb/pebble), an embedded key-value store, as the sole storage backend. - -## Reasons - -**The access pattern is a pure key-value workload.** Every read is a point lookup by `NodeID` or a sequential scan over a known key prefix (LRU bucket scan, TTL reap). SQL's query planner, index maintenance, and join execution add overhead with no benefit when every query reduces to a primary-key fetch. - -**Large blobs fragment across SQL pages.** SQL engines use fixed page sizes (typically 8–16 KB). Request and response blobs are serialised conversation items that frequently exceed one page. The SQL engine must read, assemble, and write multiple pages per blob. Pebble writes each blob as a single contiguous SSTable value; a point read recovers the whole blob in one I/O. - -**LSM write path matches the append-only workload.** Charon writes each new response as one atomic batch: one node record and one or two blob values. Pebble's LSM absorbs writes into an in-memory memtable and flushes to sorted SSTables — ideal for an append-mostly pattern where existing values are rarely updated in place. SQL B-tree pages require in-place updates and lock-based concurrency control even for pure inserts when the index must be maintained. - -**No external process.** Pebble is a pure-Go library; it embeds directly into the Charon binary. A SQL deployment requires an external database server (network dependency, separate process lifecycle, connection pooling). The single-binary deployment goal (see ADR 0001) is satisfied by an embedded store; a SQL server breaks it. - -**Pure-Go, no CGo.** Pebble cross-compiles with `CGO_ENABLED=0` to any `GOOS`/`GOARCH` target without requiring a C toolchain. SQLite requires CGo; a pure-Go SQLite binding adds significant latency overhead for the write path due to the CGo call cost per transaction. - -**The `Backend` interface abstracts the storage layer.** `chainstore.Backend` (in `internal/chainstore/backend.go`) defines `GetNode`, `GetBlobs`, `LoadChain`, `Commit`, and `Stats`. The Pebble implementation lives in `internal/chainstore/pebble/`. A DynamoDB backend is planned but not yet implemented; it would enable distributed deployments without changing chain-walk or staging logic. Switching storage implementations does not require changing chain-walk or staging logic. - -## Trade-offs Accepted - -**No ad-hoc query capability.** Debugging tools cannot run `SELECT … WHERE …` over payload content. Operational inspection requires purpose-built CLI tools (the `cache-check` CLI added in phase 5) or offline export. This is acceptable because Charon's API already exposes all necessary retrieval paths. - -**Single-writer constraint.** Pebble does not support concurrent writers from separate processes. A single Charon instance owns a Pebble directory exclusively. Horizontal scale requires sharding by response ID prefix or switching to a distributed `Backend` implementation (see the Performance and Scale section in `architecture.md`). - -**No built-in replication.** Pebble durability is limited to the local disk. WAL-based crash recovery protects against process crashes; hardware failure requires external backup or RAID. This is consistent with the single-server deployment target. diff --git a/docs/architecture.md b/docs/architecture.md index 4f3ff1c..87dae14 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -7,7 +7,7 @@ The OpenAI Responses API is *stateful from the client's perspective*: each reque Charon bridges this gap. It is an internal service that: 1. Resolves a `previous_response_id` chain into a flat, ordered `[]Item` context -2. Returns that context (plus a new response ID) to the caller before inference +2. Returns that context (plus a staging ID) to the caller before inference; the staging ID is replaced by the inference server's canonical response ID once the response is committed 3. Accepts the completed inference output for durable storage after inference Charon is **not** the client-facing API layer. It is called by a proxy that owns the @@ -21,8 +21,10 @@ client surface (e.g., `/v1/responses/` API). Client ↓ OpenAI Responses API (HTTP / SSE / WebSocket) Proxy ──────────────────────────────────────► Charon - │ POST /staging (resolve + open staging) (context resolution + storage) - │ PUT /staging/{id}/complete (store) + │ GET /responses/{id} (resolve chain) (context resolution + storage) + │ POST /responses/{id} (buffered store) + │ POST /staging (open streaming staging) + │ PUT /staging/{id}/complete (commit) │ ↓ stateless Responses API (full flat_context as input) Inference Backend (OpenAI-compatible) @@ -35,6 +37,8 @@ Proxy and Charon are **peers**: the proxy calls Charon to resolve prior context inference and to store results after inference. The proxy calls the inference backend directly — Charon is never in the inference call path. +**Why two components?** The proxy and Charon have different scaling axes and different rates of change. Proxy instances are stateless and scale horizontally without coordination. Charon scales independently based on storage throughput — a single Charon instance can serve many proxy instances. `store: false` semantics and connection-local WebSocket caches are proxy concerns; tying them to Charon would require Charon to understand connection lifecycle. Auth, TLS, and streaming protocol details change independently of storage logic and belong in the component that owns the client surface. + ### Proxy The proxy owns all client-facing concerns: @@ -44,36 +48,39 @@ The proxy owns all client-facing concerns: - Connection-local ephemeral cache for `store: false` responses (WebSocket sessions) - Request validation and routing - Streaming inference output back to the client +- Multi-tenancy: provides an optional tenant identifier with each Charon call; Charon uses it to namespace responses across tenants +- Strips non-chain-reconstruction fields from requests/responses before forwarding to Charon (Charon stores the resulting blobs opaquely) ### Charon Charon owns storage and resolution: -- Resolves `previous_response_id` chains into flat `[]Turn` context arrays via `POST /staging` -- Persists request and response blobs to durable Pebble storage +- Resolves `previous_response_id` chains into flat `[]Turn` context arrays +- Persists request and response blobs to durable storage - Manages staging records for in-flight streaming ingest; reaps orphans via background TTL worker - Runs background workers: TTL expiry, staging reaper -Charon does **not** own: SSE, WebSocket, auth, TLS, model routing, or `store: false` semantics. - --- ### Proxy–Charon interaction -The proxy calls Charon differently depending on whether this is a new chain or a continuation. +The proxy calls Charon differently depending on whether this is a new chain or a continuation, and whether the response is streamed. **New chain** (no `previous_response_id`): - No Charon resolve call. Proxy calls inference with `flat_context=[]` + `input[]`. - The inference server assigns the canonical response ID, returned in the first streaming chunk or response body. -- Proxy emits `response.created` to the client using that inference-server-assigned ID. -- If `store: true`: proxy uses the buffered `POST /responses` path (or the streaming staging protocol) to commit the response. +- If `store: true`: proxy calls `POST /responses/{canonical_id}` with the request and response blobs. -**Continuation** (has `previous_response_id`) — streaming path: +**Continuation — buffered path** (`store: true`, no streaming): +1. Proxy calls `GET /responses/{previous_response_id}` — Charon returns the resolved flat context. +2. Proxy appends new `input[]` and forwards to the inference server. +3. Proxy calls `POST /responses/{canonical_id}` with the complete request and response blobs. + +**Continuation — streaming path** (`store: true`, streaming): 1. **Open staging**: `POST /staging?prev={previous_response_id}` — Charon resolves the prior chain, creates a staging record, returns `{staging_id, flat_context[]}`. 2. **Inference**: proxy appends new `input[]` to `flat_context` and forwards to the inference server. The first streaming chunk carries the canonical response ID. -3. **Client notification**: proxy emits `response.created` using the inference-server-assigned canonical ID. -4. **Append chunks**: proxy delivers response bytes incrementally via `PUT /staging/{staging_id}/chunks/{k}`. -5. **Complete**: `PUT /staging/{staging_id}/complete?response_id={canonical_id}&total={N}` — Charon seals the staging record and commits the node atomically. +3. **Append chunks**: proxy delivers response bytes incrementally via `PUT /staging/{staging_id}/chunks/{k}`. +4. **Complete**: `PUT /staging/{staging_id}/complete?response_id={canonical_id}&total={N}` — Charon seals the staging record and commits the node atomically. If `store: false` is set, the proxy skips all Charon store calls. The `store: false` flag is a proxy-level concern; Charon is unaware of it. @@ -83,7 +90,7 @@ If `store: false` is set, the proxy skips all Charon store calls. The `store: fa ### 1. Stateless front end -The proxy holds no in-process conversation state (except the ephemeral per-connection cache for `store: false` WebSocket sessions). Any request can be served by any proxy instance. State lives exclusively in Charon's storage layer. +The proxy holds no durable state — its state is bounded to the lifetime of a single request/response (except the ephemeral per-connection cache for `store: false` WebSocket sessions). Any request can be served by any proxy instance. State lives exclusively in Charon's storage layer. The inference backend is similarly stateless. It always receives the complete flat context assembled by Charon. @@ -98,14 +105,6 @@ The storage layer uses [Pebble](https://github.com/cockroachdb/pebble), an embed The binary is identical across all levels. Only `storage.data_dir` changes. -### 3. User Inputs, not KV cache - -The storage layer persists conversation history as serialized items (text, tool calls, tool outputs). It does not attempt to persist the inference engine's KV cache. - -The expansion factor makes KV cache storage impractical (three orders of magnitude expansion). - -Alternative: Tokens are portable across model versions and inference backends and could potentially be used instead. - --- ## The Stateful-to-Stateless Translation @@ -129,19 +128,7 @@ resp_A (root) ← resp_B ← resp_C ← resp_D (head) The proxy appends the new request's `input[]` to the flat context before forwarding to the inference backend. -**Implementation strategies:** - -| Strategy | Write cost | Read cost | Storage cost | Notes | -|----------|-----------|-----------|--------------|-------| -| Entry-per-response | O(1) | O(N) — walk chain | O(N) total | Storage efficient; latency grows with chain depth | -| Full-snapshot-per-response | O(N) | O(1) — single fetch | O(N²) total | Write amplification; simplest reads | -| Checkpoint every K turns | O(1) amortized | O(K) — 1 checkpoint + ≤K deltas | O(N²/K) total | Trades storage for bounded read latency | - -Each checkpoint blob is **cumulative** — it contains all turns from chain root to its position. The checkpoint at position nK holds nK payloads. Total checkpoint storage across a chain of length N grows as K + 2K + ... + (N/K)·K ≈ N²/(2K). With K=10 and a 1000-turn chain, checkpoint storage is roughly 50× the delta storage — the cost paid for O(K) reads with a single self-contained checkpoint fetch and no chaining. - -An alternative **incremental checkpoint** design stores only the K new turns at each checkpoint position and keeps a back-reference to the prior checkpoint. This reduces storage to O(N) but requires loading ⌈N/K⌉ checkpoint blobs in sequence to reconstruct the full context — O(N/K) chained reads instead of O(K) reads. For read-heavy workloads the current cumulative design is preferable; for write-heavy workloads with very long chains the incremental design trades read latency for storage efficiency. - -Charon uses the checkpoint strategy. See [storage design](storage.md) for details. +Charon uses an entry-per-turn strategy: each response stores only its own turn, and the full context is assembled by walking the chain on each resolve call. See [storage design](storage.md) for the strategy trade-offs. --- @@ -149,14 +136,15 @@ Charon uses the checkpoint strategy. See [storage design](storage.md) for detail Charon exposes an internal HTTP API consumed only by the proxy. It is **not** required to conform to the OpenAI Responses API specification — it is designed for operational efficiency as an internal service. -### Buffered store: `POST /responses` +### Buffered store: `POST /responses/{id}` -The non-streaming path. The proxy sends a complete serialised request blob and response blob in one call. Charon resolves the previous chain, stages the request, and commits the response atomically. +The non-streaming path. The proxy first calls `GET /responses/{previous_response_id}` to resolve the prior chain, forwards to inference, then calls `POST /responses/{response_id}` with the complete request and response blobs. + +> **Open question:** `GET /responses/{id}` currently returns a single turn. A separate subresource (e.g. `GET /resolve/{id}`) may be needed to return the full assembled flat context without requiring N individual GET calls. ``` -POST /responses +POST /responses/{response_id} body: { - "response_id": "resp_...", "previous_response_id": "resp_..." | null, "tenant_key": "", "request_blob": "", @@ -164,7 +152,6 @@ body: { } Response: 201 Created X-Depth: - { "staging_id": "..." } ``` ### Streaming path: staging protocol @@ -178,13 +165,15 @@ The streaming path separates resolve from store via a three-step staging protoco 3. **Complete staging**: `PUT /staging/{id}/complete?response_id=...&total=...` — seals the staging record and commits the node into the chain store. `total` is the total chunk count. Additional staging endpoints: -- `PUT /staging/{id}/abort` — marks the staging record as aborted; no node is committed. +- `PUT /staging/{id}/abort` — marks the staging record as aborted; no node is committed. *(under review — may be removed if the TTL reaper provides sufficient cleanup)* - `GET /staging/{id}` — returns staging status (in-progress, complete, or aborted). ### Retrieve: `GET /responses/{id}` Returns the full stored record for one response — request blob, response blob, depth. No chain walk. +> **Note:** This endpoint returns a single turn's data. Resolving a full chain for inference requires either N sequential calls or a dedicated resolve endpoint (see open question above). + ### Delete: `DELETE /responses/{id}` Point delete — removes the node and its blobs. No effect on other responses in the chain. Background TTL expiry handles bulk eviction. @@ -199,7 +188,7 @@ Response IDs visible to clients are assigned by the inference server, not pre-mi **Staging ID**: A 128-bit random UUID minted by Charon at `POST /staging` time and returned to the proxy. It is never exposed to clients. It ties the resolved chain context to the subsequent chunk writes and commit. The proxy binds the canonical response ID to the staging ID on the first chunk write (`PUT /staging/{id}/chunks/0?response_id=...`). -**ID flow — continuation:** +**ID flow — continuation (streaming):** ``` POST /staging?prev={prev_response_id} → Charon returns staging_id + flat_context inference → server assigns canonical_id @@ -207,39 +196,26 @@ PUT /staging/{staging_id}/chunks/{k}?response_id={canonical_id} (first chunk bi PUT /staging/{staging_id}/complete?response_id={canonical_id}&total={N} → committed ``` -**ID flow — new chain (via buffered path):** +**ID flow — continuation (buffered):** ``` -POST /responses body: { response_id, previous_response_id=null, request_blob, response_blob } - → 201 Created; node committed atomically +GET /responses/{prev_response_id} → Charon returns flat_context +inference → server assigns canonical_id +POST /responses/{canonical_id} body: { previous_response_id, request_blob, response_blob } + → 201 Created; node committed atomically ``` ---- - -## Streaming Ingest - -The staging protocol separates chain resolution from blob commit. This allows the proxy to deliver inference output to Charon incrementally as tokens arrive, without holding all output in proxy memory. - +**ID flow — new chain:** ``` -POST /staging?prev={prevID} → staging_id, flat_context (resolve + open staging) -PUT /staging/{id}/chunks/{k} → next_expected (repeated per batch; out-of-order OK) -PUT /staging/{id}/complete?total={N} → committed (seal and write node atomically) +POST /responses/{canonical_id} body: { previous_response_id=null, request_blob, response_blob } + → 201 Created; node committed atomically ``` -Chunks are stored by offset (0-based). The proxy may write chunks from concurrent goroutines in any order; Charon assembles them in order at commit time. If the proxy crashes after `POST /staging` but before `PUT .../complete`, the staging record is reaped by the staging TTL worker (background goroutine) and no orphaned node is left in the chain. - -**Chunk size trade-offs:** - -| Chunk size | Peak proxy memory | Charon write ops | Durability boundary | -|------------|------------------|------------------|---------------------| -| 1 batch | Minimal | One per batch | Per batch | -| Full output | Full output | One commit | On completion only | - -The buffered `POST /responses` path is equivalent to a single-chunk staging flow executed atomically. - --- ## What Is and Isn't Persisted +The proxy is responsible for constructing the blobs sent to Charon — Charon stores them opaquely without parsing message content. + ### Must be persisted (for chain reconstruction) | Field | Why | @@ -266,7 +242,7 @@ The buffered `POST /responses` path is equivalent to a single-chunk staging flow ## `store: false` Semantics -When the client sets `store: false`, the proxy skips the store call to Charon after inference. The response is never written to durable storage. +When the client sets `store: false`, the proxy skips the store call to Charon after inference. The response is never written to durable storage. When `store: false`, the proxy also skips the staging open call — there is no `/staging` interaction, only the prior-chain resolve (if a `previous_response_id` is present). For WebSocket connections, the proxy maintains a connection-local in-memory cache of `store: false` responses so that `previous_response_id` lookups within the same connection still work. On disconnect, this cache is lost. A reconnecting client that references a `store: false` response ID receives `previous_response_not_found` and must start a new chain. @@ -276,20 +252,9 @@ Charon has no knowledge of `store: false`. Because write-intents are created onl ## Deployment Modes -Proxy and Charon are always separate services in production — they run in separate processes, typically on separate hosts. Colocation in the same binary is provided for testing purposes (conformance, compliance, and development iteration). - -The current storage backend is [Pebble](https://github.com/cockroachdb/pebble) — an embedded key-value store with no external process dependency. All data (chain metadata, payload blobs, LRU accounting) lives in a single Pebble database directory. - -**In-memory Pebble** (`storage.data_dir` empty — conformance and compliance testing) -- All data is lost on restart -- Suitable for running the openresponses.org compliance suite and integration tests +Proxy and Charon are separate services in production — separate processes, typically on separate hosts. -**On-disk Pebble** (`storage.data_dir` set — development and production) -- Data survives restarts; Pebble uses WAL-based crash recovery -- Single-node only: Pebble does not support multi-writer access from separate processes -- Multiple proxy instances may share one Charon process; Charon is the single source of truth for chain state - -Multi-instance Charon scaling is not currently implemented. +For testing and development, proxy and Charon may be colocated in the same binary. The proxy calls Charon's service layer directly (in-process, no HTTP hop). The binary is identical across all deployment levels; only `storage.data_dir` and service configuration change. --- @@ -297,19 +262,21 @@ Multi-instance Charon scaling is not currently implemented. ### Access patterns -Charon's workload is append-only point-lookup and sequential parent-pointer walk — no joins, no predicate scans over payload content. These patterns fit an embedded key-value store far better than a relational database. See [ADR 0004](adr/0004-kv-over-sql.md) for the full access-pattern analysis and the rationale for choosing Pebble over SQL and object-store alternatives. +Charon's workload is append-only point-lookup and sequential parent-pointer walk — no joins, no predicate scans over payload content. These patterns fit an embedded key-value store far better than a relational database. ### Single-server throughput expectations +The in-memory LRU blob cache (`chainCache`, default 64 MiB) keeps recently accessed turns in RAM; the cache-warm and cache-cold paths below reflect whether a given node's blob is cached. + These are order-of-magnitude estimates for a single Charon instance on commodity server hardware (8–32 cores, NVMe SSD). They are not benchmarks; actual numbers depend on blob size, chain depth, and hardware. -**Write throughput — new response (`POST /responses` buffered path or `PUT /staging/{id}/complete`):** +**Write throughput — new response (`POST /responses/{id}` buffered path or `PUT /staging/{id}/complete`):** Dominated by the Pebble write batch (WAL append + memtable insert). A single atomic commit writes one node record and one or two blob values. With Pebble's default write options, expect low thousands of commits per second at blob sizes of 10–100 KB. The staging flush path (`ResolveAndStage` + `StoreWithStaging`) issues two Pebble commits per turn — one for the staging node, one for the final node — which roughly halves peak write throughput versus the buffered path. **Read throughput — `GET /responses/{id}`:** `Retrieve` issues a single `GetNode` and `GetBlobs` call. No chain walk, no LRU touch. On a warm block cache, expect sub-millisecond p50 latency and tens of thousands of reads per second for typical blob sizes. -**Chain resolution latency — `POST /staging?prev=...` or `Resolve`:** +**Chain resolution latency — `POST /staging?prev=...` or `GET /responses/{id}` (full chain):** `walkAndTouch` calls `backend.LoadChain` (N sequential `GetNode` reads from leaf to root) followed by up to N `GetBlobs` calls for nodes not in the in-memory cache. Latency scales approximately linearly with chain depth N: - Cache-cold path: ~N Pebble key reads + ~N blob fetches. At 100 µs per Pebble read, a depth-10 chain resolves in roughly 2–5 ms; depth-100 in 20–50 ms. @@ -346,10 +313,8 @@ The current architecture intentionally targets single-server deployments. ## What This Design Does Not Solve -- **Durable KV cache across restarts**: The inference backend's KV cache is out of scope. This is an infrastructure-level concern orthogonal to Charon's design. - - **Semantic compaction**: Charon stores the literal content of every turn verbatim. Semantic summarization — collapsing prior turns into a shorter representation — is a proxy concern, not a Charon concern. When the proxy calls `POST /responses/compact`, it sends the turns to be compacted to the inference backend, which returns a `compaction` item with opaque `encrypted_content`. The proxy then stores this compaction item via the normal Charon store path. Charon stores the compaction item verbatim alongside the other items in the chain; it does not drop or rewrite prior entries. Which responses to compact and what to do with the resulting item are decisions made by the proxy or the calling application. -- **DAG history (branching conversations)**: The spec allows `previous_response_id` to form a DAG (two responses can share the same parent). Charon's storage design accommodates this structurally — the `chain_root_id` + `position` denormalisation and checkpoint blobs are keyed per chain, and separate branches simply produce separate keys. However, retrieving context from a non-linear DAG path is not specially optimised: each branch is walked independently, and there is no shared-prefix cache across branches. If DAG usage becomes common, branch-aware checkpoint sharing and a prefix cache would reduce redundant reads. +- **DAG history (branching conversations)**: The spec allows `previous_response_id` to form a DAG (two responses can share the same parent). Charon's storage design accommodates this structurally — separate branches produce separate node subtrees. However, retrieving context from a non-linear DAG path is not specially optimised: each branch is walked independently, and there is no shared-prefix cache across branches. -- **Chunked streaming store**: Implemented via the staging protocol. See [Streaming Ingest](#streaming-ingest). +- **Chunked streaming store**: Implemented via the staging protocol. See [Streaming Ingest](#proxy–charon-interaction) for the full flow. diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md deleted file mode 100644 index a2d6e0b..0000000 --- a/docs/implementation-plan.md +++ /dev/null @@ -1,388 +0,0 @@ -# Charon: Implementation Plan - -## Language Choice: Go - -**Go is the right choice for this workload.** The decision rests on four concrete factors: - -### 1. The workload is I/O bound, not CPU bound - -Every hot path — SQLite writes, filesystem reads, HTTP handling, zstd compression, SSE streaming — spends its time waiting on I/O. Rust's primary advantages (zero-cost abstractions, fine-grained memory control, deterministic allocation) pay dividends on CPU-bound work. For I/O-bound code that blocks on disk or network syscalls, Go and Rust produce programs with effectively identical runtime characteristics. There is no measurable performance gap for this use case. - -### 2. SQLite integration is frictionless in Go - -`modernc.org/sqlite` is pure Go — no CGo, no C toolchain dependency. It cross-compiles to any target with `GOOS`/`GOARCH` and produces a fully static binary with `CGO_ENABLED=0`. This is the single most important practical difference for the single-binary deployment goal. - -In Rust, `rusqlite` with the `bundled` feature links SQLite via the `cc` crate, which requires a C compiler at build time. Cross-compiling to a different target (e.g., building on macOS for a Linux container) requires a configured C cross-toolchain — real friction that simply does not exist in Go. - -### 3. Streaming and concurrency model fits naturally - -The background write-intent recovery worker, TTL expiry worker, and per-request chain-walk goroutines are all naturally expressed as goroutines communicating over channels, with `context.Context` for cancellation and deadlines. This maps directly onto the problem. - -The Rust equivalent — `tokio` tasks, `Arc>`, `await` across lock boundaries, fighting the borrow checker across yield points — is correct but adds ceremony to what is sequential background bookkeeping. - -### 4. The repository is already Go - -The workspace path (`go/src/github.com/elevran/charon`) establishes Go as the project language, minimizing contributor ramp-up. - -### Where Rust would be better - -Rust's `serde` + enum variants would be ideal for the discriminated union of output item types (`message`, `function_call`, `function_call_output`, `reasoning`, `compaction`). In Go, this requires an interface with type-switch or a tagged struct with `json.RawMessage` plus a second unmarshal pass — boilerplate for every variant. **Mitigation:** generate the Go types from the OpenAPI spec rather than hand-writing them. This eliminates most of the maintenance cost. - ---- - -## Architecture Summary - -``` -Proxy (separate repo) - ↓ GET /responses/{id} resolve: flat_context + new response_id - ↓ POST /responses/{id} store: write-intent + payload commit -Charon HTTP API - ↓ -Conversation Service - ↓ IndexStore interface ↓ PayloadStore interface - SQLite Local filesystem - (Phase 2: Postgres) (Phase 2: S3/MinIO) -``` - -Charon is an internal service — not client-facing. The proxy (separate repository) owns SSE, WebSocket, auth, and the OpenAI Responses API surface. Charon owns chain resolution, payload persistence, and write-intent safety. Backends are injected at startup from configuration. See [architecture.md](architecture.md) and [storage.md](storage.md) for design details. - ---- - -## Phase 1: Single Executable - -**Goal:** A working server that implements the Responses API with durable persistence, correct multi-turn history, and no external dependencies beyond the inference backend. - -**Deliverables:** -- Single binary deployable with `./charon --config config.yaml` -- IndexStore: embedded SQLite via `modernc.org/sqlite` -- PayloadStore: local filesystem (structured directory tree) -- Charon internal HTTP API (`net/http` with `chi` router): - - `GET /responses/{id}/context` — resolve: chain walk, mint response ID, return flat_context (called before inference, continuations only) - - `POST /responses/{id}` — store: write-intent + payload commit (called after inference) - - `GET /responses/{id}` — retrieve: return single stored response record (called to serve client reads) - - `DELETE /responses/{id}` — delete: point delete, no cascade - - `GET /metrics` — Prometheus metrics scrape endpoint -- Correct `previous_response_id` chain reconstruction -- Checkpoint every N turns (configurable, default 10) -- TTL expiry background worker -- Write-intent recovery background worker -- In-memory IndexStore + PayloadStore for tests -- Unit and integration test suite - -**Basic observability (Phase 1):** -- HTTP request count and latency per endpoint (`resolve`, `store`, `retrieve`, `delete`) -- Write-intent failure count (critical correctness signal) -- Chain depth at resolve time (histogram — understand usage patterns) -- Active write-intents gauge (detect stuck intents) - -**Non-goals for Phase 1:** -- SSE streaming, WebSocket (proxy concerns — separate repo) -- `/responses/compact` (proxy concern — requires inference client) -- Multi-node deployment -- Horizontal scalability -- ABAC access control (single-tenant mode) - -### Package structure - -``` -cmd/charon/ -- binary entry point, config loading, dependency wiring -internal/api/ -- HTTP handlers for Charon's internal API (resolve, store, retrieve, delete) -internal/service/ -- ConversationService: chain resolution, context assembly, checkpoint logic -internal/storage/ -- IndexStore and PayloadStore interfaces + implementations - sqlite/ -- SQLite IndexStore - filesystem/ -- Filesystem PayloadStore - memory/ -- In-memory implementations for tests -internal/model/ -- Go types for all Responses API structures - items.go -- Item discriminated union (generated from OpenAPI spec) - response.go -- ResponseRecord, ResponseMeta, Payload -internal/worker/ -- Background goroutines: TTL expiry, write-intent recovery -``` - -### Key dependencies - -| Purpose | Package | -|---------|---------| -| SQL (embedded, no CGo) | `modernc.org/sqlite` | -| SQL query builder | `github.com/jmoiron/sqlx` | -| HTTP router | `github.com/go-chi/chi/v5` | -| Metrics | `github.com/prometheus/client_golang` | -| zstd compression (optional) | `github.com/klauspost/compress/zstd` | -| UUID generation | `github.com/google/uuid` | -| Structured logging | `log/slog` (stdlib) | - -Schema initialization uses `CREATE TABLE IF NOT EXISTS` at startup — no migration framework. Since deployments always start with empty stores, live schema migration is not a requirement. - -Phase 2 additions: `aws-sdk-go-v2` (S3/MinIO PayloadStore), `pgx` (PostgreSQL IndexStore), `github.com/hashicorp/golang-lru/v2` (hot-path context cache in Charon). - -### Data directory layout (filesystem PayloadStore) - -``` -{data_dir}/ - responses.db ← SQLite database - payloads/ - {chain_root_id}/ - {position:08d}_{response_id}.json.zst ← individual response payloads - checkpoint_{position:08d}_{response_id}.json.zst ← checkpoint blobs -``` - -Using `chain_root_id` + `position` in the path means the full chain is enumerable by directory listing, independent of SQL, and checkpoint files are co-located with their chain's individual payloads. - -### Configuration - -```yaml -server: - listen: ":8080" - -inference: - base_url: "http://localhost:11434/v1" # vLLM, Ollama, or any OpenAI-compatible endpoint - api_key: "" # optional; passed as Authorization header - -storage: - index: "sqlite" # sqlite | postgres - payload: "filesystem" # filesystem | s3 - data_dir: "./data" # used by sqlite and filesystem backends - checkpoint_interval: 10 # create a checkpoint every N turns - ttl_days: 30 # responses expire after N days - -sqlite: - path: "./data/responses.db" # overrides data_dir if set - wal_mode: true - busy_timeout_ms: 5000 - -# Phase 2 only: -postgres: - dsn: "" -s3: - endpoint: "" - bucket: "" - access_key: "" - secret_key: "" -``` - -### Response ID format - -``` -resp_<32-char hex UUID without dashes> ← canonical; assigned by the inference server -rsrv_<32-char hex UUID without dashes> ← reservation; assigned by Charon at resolve time -``` - -Examples: -- Canonical: `resp_4a3f8c2e1b0d9f7a6e5c4b3a2d1e0f9c` — the ID clients see and Charon stores as the primary key -- Reservation: `rsrv_9f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c` — proxy-internal, never exposed to clients - -**Canonical IDs** are assigned by the inference server (vLLM or equivalent) and returned in the first streaming chunk or response body. When the inference backend returns IDs in a different format (e.g. `chatcmpl-` for chat completions), the proxy translates them to `resp_` format using `uuid.New()` keyed from the backend ID, or simply re-prefixes. IDs are opaque — no embedded timestamps, user identifiers, or model identifiers. - -**Reservation IDs** are minted by Charon at resolve time using `uuid.New().String()` stripped of dashes, prefixed with `rsrv_`. They serve as correlation handles between the resolve call and the subsequent store call, and as a fallback response ID if inference fails before any canonical ID is known. - -### Write path - -The proxy makes two calls to Charon per inference turn (resolve is skipped for new chains): - -**Resolve (before inference, continuations only):** -``` -GET /responses/{previous_response_id}/context // proxy → Charon - ↓ -ConversationService.Resolve(ctx, previousID) - ↓ - walk chain with checkpoint: IndexStore + PayloadStore - → returns flat []Item context (history only) + reservation_id - ↓ -proxy appends new input[] to flat_context -proxy forwards flat_context + input[] to inference backend - ↓ (first streaming chunk from inference server) -proxy extracts canonical_id from inference server response -proxy sends response.created to client with canonical_id -``` - -**Store (after inference, Phase 1: fully buffered):** -``` -POST /responses/{canonical_id} // proxy → Charon -body: { reservation_id, previous_response_id, input[], output[], usage, status } - ↓ -ConversationService.Store(ctx, canonicalID, req) - ↓ - INSERT write_intents (phase=pending, reservation_id, response_id=canonical_id) - PayloadStore.Put(ctx, payloadKey, payload) // atomic: write-to-temp, rename - UPDATE write_intents (phase=file_written) - IndexStore.Put(ctx, meta) - UPDATE write_intents (phase=committed) -``` - -For new chains, `previous_response_id` is null and `reservation_id` is omitted. `store: false` requests skip the store call entirely. No write-intent is created. - -**Store (after inference, future: chunked streaming):** - -See [Streaming Store Modes](architecture.md#streaming-store-modes). The chunked protocol opens the write-intent at stream start and commits at stream close. The write path for individual chunks is: -``` -POST /responses/{canonical_id} // open stream → INSERT write_intents (phase=stream_open) -PATCH /responses/{canonical_id} { items: [...] } // each chunk → accumulate -PATCH /responses/{canonical_id} { items: [...], usage, status } // final → PayloadStore.Put, IndexStore.Put, UPDATE write_intents (phase=committed) -``` - -### Write-intent safety (Phase 1, filesystem) - -Write-intents are created only at store time. If the proxy crashes before calling store (e.g., inference failed with no failure notification, or `store: false`), no orphaned state exists. - -For the SQLite + filesystem backend, the write sequence within a store call: - -1. INSERT `write_intents` row with `phase = 'pending'`, `response_id = canonical_id`, `reservation_id` (if continuation), `created_at = updated_at = now` -2. Write payload file (atomic: write-to-temp, `os.Rename`) -3. UPDATE `write_intents` SET `phase = 'file_written'`, `updated_at = now` -4. INSERT `responses` metadata row -5. UPDATE `write_intents` SET `phase = 'committed'`, `updated_at = now` - -Recovery condition: `phase IN ('pending', 'file_written') AND updated_at < now - stale_threshold` (default 5 minutes). On startup and periodically, the recovery worker finds stale intents and either completes them (re-write is safe: deterministic payload key) or marks them `failed`. - -### Success criteria for Phase 1 - -- All conformance test cases pass: `resolve-new-chain`, `resolve-multi-turn`, `resolve-with-checkpoint`, `store-and-retrieve`, `write-intent-recovery`, `ttl-expiry` -- A conversation of 100 turns reconstructs correctly with correct flat_context returned -- Single binary starts with zero dependencies: `./charon --config config.yaml` -- Unit and integration tests cover chain reconstruction, checkpoint logic, and write-intent recovery at each failure phase -- Estimated code size: ~2,000–3,000 lines (no proxy/SSE/WebSocket scope) - ---- - -## Phase 2: Scaled Storage Backends - -**Goal:** Multiple API instances sharing a common storage backend. No API or business logic changes — only backend implementations added and wired into the existing interfaces. - -**Deliverables:** -- PostgreSQL IndexStore implementation -- S3/MinIO PayloadStore implementation (using `aws-sdk-go-v2` with a custom endpoint for MinIO) -- Write-intent recovery with PostgreSQL as the coordinator -- Connection pooling configuration - -**Deployment path for Phase 2:** - -Phase 1 and Phase 2 are independent deployments chosen based on scale and operational requirements — there is no data migration between them. Both always start with empty index and payload stores. - -1. Deploy Charon with PostgreSQL + S3/MinIO config -2. Set `storage.index` to `postgres`, configure `postgres.dsn` -3. Set `storage.payload` to `s3`, configure bucket and credentials -4. Application code is identical to Phase 1 — only configuration changes - -**Success criteria for Phase 2:** - -- Two API instances can serve the same conversation concurrently without corruption -- Write-intent recovery correctly completes or fails partial writes after a simulated crash -- Full chain retrieval for a 500-turn conversation completes in under 500ms - ---- - -## Phase 3: Proxy Layer and Compliance Testing - -**Goal:** Expose the client-facing OpenAI Responses API surface and validate it against the [openresponses.org compliance suite](https://www.openresponses.org/compliance) running entirely locally — no remote API, no GPU required for CI. - -Charon is an internal service; it never speaks the Responses API directly. This phase builds the thin proxy layer that sits in front of Charon and translates the client-facing protocol into Charon's resolve/store calls. Once the proxy exists, the compliance suite can be pointed at `http://localhost:PORT` and run headlessly. - -**Deliverables:** - -*Proxy layer (client-facing HTTP):* -- `POST /responses` (REST) — resolves chain if `previous_response_id` present, forwards to inference, stores result -- `POST /responses` with `stream: true` — same flow, but streams SSE events to the client as tokens arrive from the inference backend -- `WebSocket /responses` — accepts `response.create` messages, streams output events back; supports `store: false` with connection-local cache -- `POST /responses/compact` — forwards to inference for compaction; stores result via Charon -- `GET /responses/{id}`, `DELETE /responses/{id}` — thin pass-through to Charon's retrieve/delete endpoints - -*Mock inference backend (for CI):* -- A deterministic local HTTP server that implements the OpenAI chat completions API -- Returns canned responses matched against input patterns (no GPU, no network) -- Sufficient to satisfy the compliance suite's content assertions (e.g. "say hello in exactly 3 words") -- Configurable via the existing `inference.base_url` config key — no code changes needed to swap in the real inference backend - -*Compliance integration:* -- `bun run test:compliance --base-url http://localhost:PORT` runnable locally and in CI -- CI runs the 12 achievable tests (those not requiring real LLM reasoning); remaining 5 are gated behind a `--extended` flag for use against a real deployment - -**Test feasibility breakdown:** - -| Category | Tests | CI target | -|----------|-------|-----------| -| Local fixture only | `response-output-phase-schema` | ✓ | -| Error / missing-data only | `compact-missing-model`, `websocket-previous-response-not-found`, `websocket-reconnect-store-false-recovery`, `websocket-failed-continuation-evicts-cache` | ✓ | -| Deterministic mock sufficient | `basic-response`, `streaming-response`, `system-prompt`, `multi-turn`, `websocket-response`, `websocket-sequential-responses`, `websocket-continuation` | ✓ | -| Requires real LLM | `tool-calling`, `image-input`, `assistant-phase`, `compact-response`, `websocket-compact-new-chain` | extended only | - -**Infrastructure note:** The compliance runner requires `bun` (TypeScript runtime). CI must install it (`curl -fsSL https://bun.sh/install | bash` or the distro package). The runner is cloned from [github.com/openresponses/openresponses](https://github.com/openresponses/openresponses) and invoked as `bun run test:compliance`. - -**Deployment model:** For the single-binary deployment, the proxy and Charon are colocated in the same process. The proxy calls Charon's service layer directly (in-process function calls, no HTTP hop). For multi-instance deployments (Phase 2 backends), the proxy makes real HTTP calls to a separate Charon instance. - -**Success criteria for Phase 3:** - -- All 12 CI-targeted compliance tests pass against the local stack with the mock inference backend -- `bun run test:compliance --base-url http://localhost:PORT` exits 0 in CI -- SSE streaming delivers `response.created`, one or more `response.output_item.added` events, and `response.completed` in that order -- `store: false` responses are served within the same WebSocket connection but not retrievable after reconnect (`previous_response_not_found`) -- Swapping `inference.base_url` from mock to a real vLLM endpoint requires zero code changes - ---- - -## Phase 4: Multi-Tenant and Access Control - -**Goal:** Tenant isolation, per-user response scoping, authentication. - -**Deliverables:** -- ABAC attribute model: `owner_principal`, `roles`, `teams`, `projects`, `namespaces` -- Authentication middleware (header-based for trusted gateways; bearer token for direct access) -- Per-request access check on all read and delete operations -- `AuthorizedIndexStore` wrapper that injects WHERE clauses based on requester attributes -- Audit log for access denials - -**Design notes:** - -The access model must avoid the pitfall identified in existing implementations: a row should be accessible to the owner OR to users explicitly granted access — not to anyone who shares any team value. Shared access should be opt-in (explicit grants), not opt-out (attribute overlap). - -**Success criteria for Phase 4:** - -- User A cannot read User B's responses -- Shared responses (explicitly shared by owner) are accessible to designated users -- Unauthenticated requests in authenticated mode return 401, not 200 - ---- - -## Phase 5: Observability and Operational Hardening - -**Goal:** Production-grade operations: metrics, structured logging, graceful shutdown, rate limiting. - -**Deliverables:** -- Extended Prometheus metrics: checkpoint write rate and size, TTL expiry rate, storage backend latency, background worker run duration -- `GET /healthz` and `GET /readyz` probes -- Graceful shutdown: complete in-flight store calls before exit, allow background workers to finish their current iteration -- Structured logging hardening: audit all log call sites, ensure no raw content at INFO level - -**Operational tooling:** - -- `charon reconcile` subcommand: run the write-intent recovery and object store reconciliation job on demand - ---- - -## Deferred / Out of Scope - -The following are explicitly out of scope for initial delivery. Documenting them here prevents scope creep. - -**KV cache persistence:** As established in the architecture document, storing transformer KV cache is not planned. The expansion factor (30,000–160,000×) makes it impractical in durable storage. Inference backends handle KV cache in GPU memory. - -**Compaction endpoint:** `/responses/compact` is a proxy concern — it requires calling an inference backend to produce the compaction summary. Charon's role is standard chain resolution (resolve call); the proxy drives the compact flow. Automatic background summarization is similarly out of scope for Charon. - -**Segment packing (Option D storage):** The pack-file approach offers the best long-term storage efficiency but is significantly more complex to implement. It becomes relevant if storage costs are measurable at scale. Option B + Option C snapshotting is the target for Phase 1 and 2. - -**Embedding / vector search:** This is a stateful serving layer for structured conversation history, not a retrieval-augmented generation system. Vector indexing of response content is out of scope. - -**DAG optimization:** The spec allows `previous_response_id` to form a DAG (two responses can share the same parent). The storage design accommodates this structurally, but tree-shaped retrieval is not specially optimized. It falls back to standard chain walk from each leaf. - -**Chunked streaming store:** Phase 1 uses fully-buffered mode. Chunked delivery to Charon — from N-token batches down to single-token granularity — improves recovery granularity and reduces peak proxy memory, but requires a streaming ingest protocol in Charon (`PATCH` per chunk or ndjson over chunked HTTP) plus additional write-intent phases (`stream_open`). The design is specified in [Streaming Store Modes](architecture.md#streaming-store-modes); implementation is deferred to post-Phase-1. - ---- - -## Estimated Scope - -| Phase | Estimated LOC | Key complexity | -|-------|--------------|----------------| -| Phase 1 (single binary) | 2,500 – 4,000 | Chain reconstruction, checkpoint logic, write-intent recovery | -| Phase 2 (scaled backends) | +800 – 1,500 | PostgreSQL adapter, S3 adapter, cross-system consistency | -| Phase 3 (proxy + compliance) | +1,500 – 2,500 | SSE streaming, WebSocket, mock inference backend, compliance wiring | -| Phase 4 (access control) | +600 – 1,000 | ABAC model, middleware, authorized store wrapper | -| Phase 5 (observability) | +400 – 700 | Metrics, structured logging, operational subcommands | - -The dominant complexity is not in routing or storage glue — it is in **managing history state correctly over time**: checkpoint boundaries, write-intent recovery, TTL, `store: false` semantics, and the `encrypted_content` round-trip for reasoning and compaction items. These correctness concerns are where most bugs will occur and where the test suite must be most thorough. diff --git a/docs/storage.md b/docs/storage.md index b35b9c4..24b3220 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -25,88 +25,54 @@ There are no range scans over content. The hot path is: node lookup → chain wa --- -## Storage Backend: Pebble +## Why a Key-Value Store? -Charon uses [Pebble](https://github.com/cockroachdb/pebble), an embedded LSM-tree key-value store. All data lives in a single Pebble database directory. There are no external processes. +Charon's access patterns are: +- **Append-only writes**: one new node per turn, committed atomically +- **Point lookups**: fetch a node by its response ID +- **Sequential parent-pointer walks**: N sequential node reads from leaf to root during resolve +- **Background eviction**: scan LRU-ordered keys to drop the oldest entries -When `storage.data_dir` is empty or omitted, Charon opens an in-memory Pebble instance — data is lost on restart. Use this for conformance/compliance testing and CI. +These patterns — no joins, no predicate scans over content — fit a key-value store far better than a relational database. An embedded KV store (Pebble) also removes the operational overhead of an external database process and allows the binary to run with no external dependencies. -### Key Schema +--- -All keys are prefixed by a single type byte: +## Chain Reconstruction Strategy -| Prefix | Key | Value | Purpose | -|--------|-----|-------|---------| -| `0x01` | `metaKey(nodeID)` | binary-encoded Node | Per-node metadata: parent, blob ref, position, timestamps | -| `0x02` | `blobKey(blobID)` | raw payload bytes | Input + output items for a single node | -| `0x03` | `lruKey(bucket, nodeID)` | empty | LRU ordering for capacity eviction | -| `0x04` | `childKey(parent, child)` | empty | Forward child index for chain walks | -| `0x05` | `statsKey` | counters | Total entry count and byte usage | -| `0x06` | `responseIDKey(nodeID)` | caller-supplied response ID string | Maps internal NodeID to external response ID | -| `0x07` | `stagingKey(stagingID)` | partial Node | In-progress streaming nodes (invisible to chain walks) | +| Strategy | Write cost | Read cost | Storage cost | +|----------|-----------|-----------|--------------| +| Entry-per-turn | O(1) | O(N) — walk chain | O(N) total | +| Full-snapshot | O(N) | O(1) — single fetch | O(N²) total | -NodeIDs are 20-byte opaque identifiers assigned internally. BlobIDs are 16-byte identifiers derived from the blob content. All writes to a node (meta + blob + LRU + child + response ID) are committed in a single atomic Pebble batch. +Charon uses **entry-per-turn**: each response stores only its own turn. Chain reconstruction walks from head to root on each resolve. This is storage-efficient and write-fast; read latency grows linearly with chain depth. --- -## Payload Schema +## Storage Backend: Pebble -Each node's blob is the JSON-serialized content required for chain reconstruction: +Charon uses [Pebble](https://github.com/cockroachdb/pebble), an embedded LSM-tree key-value store. All data lives in a single Pebble database directory. There are no external processes. -```go -type ResponsePayload struct { - ID string - PreviousResponseID *string - InputItems []Item // what the caller sent this turn - OutputItems []Item // what the model produced this turn -} -``` +When `storage.data_dir` is empty or omitted, Charon opens an in-memory Pebble instance — data is lost on restart. Use this for conformance/compliance testing and CI. + +### What Is Stored -`Item` is a discriminated union covering: +Charon stores four categories of objects in the KV database: **node metadata** (per-turn chain linkage and timestamps), **payload blobs** (the request and response content for each turn), an **LRU ordering index** (bucket-ordered keys used for capacity eviction), and **staging records** (transient in-flight streaming nodes that are invisible to chain walks and promoted to full nodes on stream commit). -| Type | Description | -|------|-------------| -| `message` | User/assistant/system text | -| `function_call` | Tool call generated by the model | -| `function_call_output` | Tool result provided by the caller | -| `reasoning` | Opaque reasoning blob with `encrypted_content` | -| `compaction` | Opaque compaction blob with `encrypted_content` | +--- -`encrypted_content` fields are opaque provider blobs. They must be stored and returned verbatim. +## Payload Content -**Fields stored for API completeness but NOT used for reconstruction:** -`status`, `created_at`, `model`, `usage`, `error`, `temperature`, `top_p`, `tools`, `tool_choice`, `instructions`, `metadata`, `service_tier`. +Each node's blob contains the items needed for chain reconstruction: the input and output items for that turn, plus a `previous_response_id` back-pointer. Fields not used for reconstruction — sampling params, tools, instructions, operational metadata — are stored for API completeness but not consulted during resolve. `instructions` is explicitly excluded from the reconstructed context; it is re-supplied by the caller on each turn. -`instructions` is explicitly excluded from the reconstructed context — it is re-supplied by the caller on each turn. +`encrypted_content` fields on reasoning and compaction items are opaque provider blobs. They must be stored and returned verbatim. --- ## Chain Resolution -Charon walks the parent-pointer chain from the head response back to the root, collecting input and output items at each node. The assembled flat context is returned to the proxy before inference. - -```go -// Simplified resolution pseudocode -func Resolve(headID string) ([]Item, error) { - var nodes []*Node - id := headID - for id != "" { - n, _ := store.GetByResponseID(id) - nodes = append(nodes, n) - id = n.ParentResponseID - } - slices.Reverse(nodes) - var items []Item - for _, n := range nodes { - payload := store.GetBlob(n.BlobID) - items = append(items, payload.InputItems...) - items = append(items, payload.OutputItems...) - } - return items, nil -} -``` +Resolution walks the parent-pointer chain from the head response back to the root, collecting input and output items at each node in reverse order, then reverses the list to produce chronological order. The assembled flat context is returned to the proxy before inference. -**Depth and context caps** — resolution aborts early with a structured error if: +Resolution aborts early with a structured error if: - The chain walk exceeds `storage.max_chain_depth` hops → `chain_too_deep` (HTTP 422) - The assembled context exceeds `storage.max_context_bytes` bytes → `context_too_large` (HTTP 422) @@ -122,7 +88,7 @@ Responses expire after `storage.ttl_days` days (default 30). A background TTL re When `storage.max_responses` or `storage.max_payload` limits are set, Charon evicts the oldest chains first (LRU order) whenever a new node would exceed the cap. -The LRU index (`0x03` keys) sorts nodes into time buckets. Eviction scans the oldest bucket and removes entire chains (all nodes sharing the same chain root) to preserve chain coherence — partial eviction of a chain would leave orphaned nodes. +The LRU index sorts nodes into time buckets. Eviction scans the oldest bucket and removes entire chains (all nodes sharing the same chain root) to preserve chain coherence — partial eviction of a chain would leave orphaned nodes. ### Deletion @@ -132,21 +98,17 @@ The LRU index (`0x03` keys) sorts nodes into time buckets. Eviction scans the ol ## `store: false` Semantics -When the client sets `store: false`, the proxy skips the store call to Charon after inference. The response is never written to durable storage. - -For WebSocket connections, the proxy maintains a connection-local in-memory cache of `store: false` responses so that `previous_response_id` lookups within the same connection still work. On disconnect, this cache is lost. - -Charon has no knowledge of `store: false`. No orphaned state is left in Charon when a minted reservation ID never receives a store call. +Charon has no knowledge of `store: false`; the proxy simply skips the store call. --- ## Staging (Streaming Ingest) -Streaming ingest (Phase 6) delivers output items to Charon in chunks as tokens arrive from the inference backend. During streaming, the node is written to a staging slot (`0x07` prefix) — invisible to chain walks and resolve calls. On stream commit, the staging record is atomically promoted to a full node. +Streaming ingest delivers output items to Charon in chunks as tokens arrive from the inference backend. During streaming, the node is written to a staging slot — invisible to chain walks and resolve calls. On stream commit, the staging record is atomically promoted to a full node. The background reaper removes staging records older than `StagingTTL` (default 1h) to recover from abandoned streams. -See [architecture docs](architecture.md#streaming-store-modes) for the full streaming protocol. +See [architecture docs](architecture.md#proxy–charon-interaction) for the full streaming protocol. --- @@ -154,8 +116,6 @@ See [architecture docs](architecture.md#streaming-store-modes) for the full stre - **Multi-instance deployments**: Pebble is single-process. Horizontal scaling of Charon requires a distributed backend. This is not yet implemented. -- **Durable KV cache across restarts**: The inference backend's KV cache is out of scope. - - **Semantic compaction**: Charon stores literal content verbatim. The proxy handles compaction decisions and stores the resulting `compaction` item via the normal store path. -- **DAG history (branching conversations)**: The spec allows `previous_response_id` to form a DAG. Charon's child index (`0x04` prefix) accommodates this structurally — branches produce separate node subtrees. Resolution walks each branch independently with no shared-prefix optimisation. +- **DAG history (branching conversations)**: The spec allows `previous_response_id` to form a DAG. Charon's storage accommodates this structurally — branches produce separate node subtrees. Resolution walks each branch independently with no shared-prefix optimisation. diff --git a/docs/terminology.md b/docs/terminology.md deleted file mode 100644 index 7e72577..0000000 --- a/docs/terminology.md +++ /dev/null @@ -1,67 +0,0 @@ -# Charon: Bounded Context - -## Glossary - -### Charon - -The internal storage and resolution service. Charon is **not** client-facing. It accepts calls from the proxy, resolves `previous_response_id` chains into flat `[]Item` context arrays, and persists completed response payloads to durable storage. - -Charon does not own: SSE streaming, WebSocket handling, authentication, TLS, model routing, or `store: false` semantics. - -### Proxy - -The client-facing component that owns the OpenAI Responses API surface. The proxy handles HTTP transport (REST, SSE, WebSocket), authentication, TLS termination, and streaming. - -For new chains (no `previous_response_id`), the proxy calls Charon's buffered store (`POST /responses`) after inference completes. For continuations, it calls `POST /staging` to resolve before inference (receiving a `staging_id` and `flat_context`) and uses the staging protocol to store incrementally after. If `store: false`, it skips the store call entirely. - -The proxy and Charon may be colocated in a single binary (Phase 1) or deployed as separate services (Phase 2+). - -### Resolve - -The operation Charon performs when the proxy calls `POST /staging?prev={previousID}`. Charon walks the stored chain, assembles the flat `[]Turn` context from prior turns, creates a staging record, and returns `{staging_id, flat_context, turns}` to the proxy. The proxy uses `flat_context` to build the inference request. Only called for continuations — new chains use the buffered `POST /responses` path after inference. - -### Staging ID - -A 128-bit random UUID minted by Charon at `POST /staging` time. It is the correlation handle for the streaming ingest protocol. Never exposed to clients. The proxy binds the canonical response ID to the staging ID on the first chunk write and uses it through `PUT /staging/{id}/complete`. - -### Store - -The operation the proxy performs after inference completes. Two paths: -- **Buffered** (`POST /responses`): sends complete request and response blobs in one call. Atomic. -- **Streaming** (`PUT /staging/{id}/chunks/{k}` + `PUT /staging/{id}/complete`): delivers chunks as inference streams, then commits. Skipped entirely when `store: false`. - -### Charon API - -Charon's HTTP API is an internal contract between the proxy and Charon. It is not required to conform to the OpenAI Responses API specification and is designed for operational efficiency. The proxy is responsible for mapping between Charon's internal API and the client-facing Responses API surface. - -### Response - -A single turn in a conversation: one request blob + one response blob, linked to the preceding turn via `previous_response_id`. Stored as a Node (metadata) plus blob entries in Pebble. - -### Chain - -A singly-linked list of responses connected via `previous_response_id`. Charon walks the chain during resolve to produce a flat ordered context. Chain root = the first response with no `previous_response_id`. - -### Checkpoint - -A materialized snapshot of the full flat context at a specific position in the chain, written every K turns (configurable). Bounds chain-walk cost to O(K) instead of O(N). See [storage.md](storage.md). - -### Write-Intent - -A staging record created before blobs are committed, used to detect and recover from partial writes when the process crashes mid-streaming. Orphaned staging records are reaped by the background staging TTL worker. See [storage.md](storage.md) for the recovery protocol. - -### flat_context - -The ordered `[]Item` array that Charon returns from a resolve call. Contains all input and output items from the resolved chain, in chronological order, ready to be forwarded to the inference backend (after the proxy appends the new `input[]`). - -### `store: false` - -A client-set flag indicating that the response must not be written to durable storage. The proxy skips the store call to Charon. For WebSocket sessions, the proxy maintains a connection-local in-memory cache of `store: false` responses to serve within-connection `previous_response_id` lookups. Charon is unaware of `store: false`. - -### Instructions - -The system prompt supplied per-request by the client. Explicitly excluded from stored history by spec design — re-injected each request — so callers can change the system prompt mid-conversation without starting a new chain. - -### Inference Backend - -A stateless OpenAI-compatible HTTP endpoint (vLLM, Ollama, etc.). Always receives the complete flat context. Has no concept of sessions or chains. Assigns the **canonical response ID** returned in the first streaming chunk or response body — this is the ID Charon stores as the primary key and clients receive in `response.created`. When the backend uses a non-`resp_` ID format (e.g. `chatcmpl-`), the proxy translates it before forwarding to the client and to Charon. From dd4816a44aba6f4533b50c18db5d9e8e8b81b077 Mon Sep 17 00:00:00 2001 From: Etai Lev Ran Date: Fri, 10 Jul 2026 09:50:15 +0300 Subject: [PATCH 2/2] docs: fix API accuracy in architecture.md after PRs #87 and #88 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All store paths (new chain and continuation) now use the staging protocol: POST /staging → PUT chunks → PUT complete. The old GET /responses/{id} (resolve chain) + POST /responses/{id} (buffered store) flows no longer reflect what the proxy actually does. Specific fixes: - System diagram: replace wrong Charon calls with actual endpoints - Proxy-Charon interaction: unify new chain and continuation under staging protocol; document store:false path (GET /chain/{id}, added in PR #87) - Charon API section: rename buffered-store section to staging protocol (primary path), add GET /chain/{id} section, fix POST /responses body fields (prev_id/response_id/request_blob/response_blob + X-Tenant-Key header, not previous_response_id/tenant_key in body) - Remove stale 'open question' note (answered by GET /chain/{id}) - ID flow diagrams: collapse to one staging-protocol diagram; remove wrong GET /responses (chain) + POST /responses/{id} buffered diagrams - store:false semantics: describe GET /chain/{id} for continuations - Performance section: update endpoint names to match current API --- docs/architecture.md | 117 +++++++++++++++++++++---------------------- 1 file changed, 56 insertions(+), 61 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 87dae14..f4e6ce0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -21,10 +21,11 @@ client surface (e.g., `/v1/responses/` API). Client ↓ OpenAI Responses API (HTTP / SSE / WebSocket) Proxy ──────────────────────────────────────► Charon - │ GET /responses/{id} (resolve chain) (context resolution + storage) - │ POST /responses/{id} (buffered store) - │ POST /staging (open streaming staging) + │ POST /staging (open staging, resolve chain) (context resolution + storage) + │ PUT /staging/{id}/chunks/{k} (append chunk) │ PUT /staging/{id}/complete (commit) + │ GET /chain/{id} (read-only chain fetch, no commit) + │ GET /responses/{id} (point retrieve) │ ↓ stateless Responses API (full flat_context as input) Inference Backend (OpenAI-compatible) @@ -64,25 +65,24 @@ Charon owns storage and resolution: ### Proxy–Charon interaction -The proxy calls Charon differently depending on whether this is a new chain or a continuation, and whether the response is streamed. +The proxy calls Charon differently depending on whether the response will be stored and whether the turn has a prior context to fetch. -**New chain** (no `previous_response_id`): -- No Charon resolve call. Proxy calls inference with `flat_context=[]` + `input[]`. -- The inference server assigns the canonical response ID, returned in the first streaming chunk or response body. -- If `store: true`: proxy calls `POST /responses/{canonical_id}` with the request and response blobs. +**New chain** (`store: true`, no `previous_response_id`): +1. **Open staging**: `POST /staging` (no `prev` param) — Charon creates a staging record and returns a `staging_id`; `flat_context` is empty. +2. **Inference**: proxy calls inference with `flat_context=[]` + `input[]`. The inference server assigns the canonical response ID. +3. **Append chunks**: proxy delivers response bytes via `PUT /staging/{staging_id}/chunks/{k}`. +4. **Complete**: `PUT /staging/{staging_id}/complete?response_id={canonical_id}&total={N}` — Charon commits the node. -**Continuation — buffered path** (`store: true`, no streaming): -1. Proxy calls `GET /responses/{previous_response_id}` — Charon returns the resolved flat context. -2. Proxy appends new `input[]` and forwards to the inference server. -3. Proxy calls `POST /responses/{canonical_id}` with the complete request and response blobs. - -**Continuation — streaming path** (`store: true`, streaming): +**Continuation** (`store: true`, `previous_response_id` present): 1. **Open staging**: `POST /staging?prev={previous_response_id}` — Charon resolves the prior chain, creates a staging record, returns `{staging_id, flat_context[]}`. -2. **Inference**: proxy appends new `input[]` to `flat_context` and forwards to the inference server. The first streaming chunk carries the canonical response ID. -3. **Append chunks**: proxy delivers response bytes incrementally via `PUT /staging/{staging_id}/chunks/{k}`. +2. **Inference**: proxy appends new `input[]` to `flat_context` and forwards to the inference server. The inference server assigns the canonical response ID. +3. **Append chunks**: proxy delivers response bytes via `PUT /staging/{staging_id}/chunks/{k}`. 4. **Complete**: `PUT /staging/{staging_id}/complete?response_id={canonical_id}&total={N}` — Charon seals the staging record and commits the node atomically. -If `store: false` is set, the proxy skips all Charon store calls. The `store: false` flag is a proxy-level concern; Charon is unaware of it. +**`store: false`** (no persistence): +- The proxy skips the staging open call entirely. +- If `previous_response_id` is present, the proxy fetches context via `GET /chain/{previous_response_id}` (read-only, no staging record opened). +- No chunks are written and no commit is issued. --- @@ -136,48 +136,49 @@ Charon uses an entry-per-turn strategy: each response stores only its own turn, Charon exposes an internal HTTP API consumed only by the proxy. It is **not** required to conform to the OpenAI Responses API specification — it is designed for operational efficiency as an internal service. -### Buffered store: `POST /responses/{id}` +### Staging protocol: `POST /staging` → chunks → complete -The non-streaming path. The proxy first calls `GET /responses/{previous_response_id}` to resolve the prior chain, forwards to inference, then calls `POST /responses/{response_id}` with the complete request and response blobs. +All store paths (new chain and continuation) use the staging protocol: -> **Open question:** `GET /responses/{id}` currently returns a single turn. A separate subresource (e.g. `GET /resolve/{id}`) may be needed to return the full assembled flat context without requiring N individual GET calls. +1. **Open staging**: `POST /staging?prev={prevID}` — resolves the prior chain (if `prev` is present), creates a staging record, returns `{staging_id, turns[]}`. -``` -POST /responses/{response_id} -body: { - "previous_response_id": "resp_..." | null, - "tenant_key": "", - "request_blob": "", - "response_blob": "" -} -Response: 201 Created - X-Depth: -``` +2. **Append chunks**: `PUT /staging/{id}/chunks/{k}` — delivers one batch of response bytes (0-based index `k`). The first chunk call binds the canonical response ID via `?response_id=...`. -### Streaming path: staging protocol - -The streaming path separates resolve from store via a three-step staging protocol, allowing the proxy to begin inference immediately after resolve and deliver response chunks incrementally: - -1. **Open staging**: `POST /staging?prev={prevID}` — resolves the prior chain, creates a staging record, returns a `staging_id`. The `flat_context` assembled here is included in the response for the proxy to use when building the inference request. - -2. **Append chunks**: `PUT /staging/{id}/chunks/{k}` — delivers one batch of response bytes (0-based offset `k`). Returns next expected offset. Chunks may arrive out of order; Charon sorts at commit time. - -3. **Complete staging**: `PUT /staging/{id}/complete?response_id=...&total=...` — seals the staging record and commits the node into the chain store. `total` is the total chunk count. +3. **Complete staging**: `PUT /staging/{id}/complete?response_id=...&total=...` — seals the staging record and commits the node into the chain store. `total` is the total number of chunks. Additional staging endpoints: - `PUT /staging/{id}/abort` — marks the staging record as aborted; no node is committed. *(under review — may be removed if the TTL reaper provides sufficient cleanup)* - `GET /staging/{id}` — returns staging status (in-progress, complete, or aborted). -### Retrieve: `GET /responses/{id}` +### Read-only chain fetch: `GET /chain/{id}` + +Returns the full assembled flat context for the chain rooted at `{id}` without opening a staging record or committing the current turn's request blob. Used by the proxy for `store: false` continuations. -Returns the full stored record for one response — request blob, response blob, depth. No chain walk. +### Retrieve: `GET /responses/{id}` -> **Note:** This endpoint returns a single turn's data. Resolving a full chain for inference requires either N sequential calls or a dedicated resolve endpoint (see open question above). +Returns the stored record for one response — request blob, response blob, depth. No chain walk. ### Delete: `DELETE /responses/{id}` Point delete — removes the node and its blobs. No effect on other responses in the chain. Background TTL expiry handles bulk eviction. +### Buffered store: `POST /responses` + +Commits a complete turn (both request and response blobs) in a single call. Used in tests; in production the proxy uses the staging protocol for all paths. + +``` +POST /responses +X-Tenant-Key: +body: { + "prev_id": "resp_..." | , + "response_id": "resp_..." | , + "request_blob": "", + "response_blob": "" +} +Response: 201 Created + X-Depth: +``` + --- ## Response ID Lifecycle @@ -188,26 +189,18 @@ Response IDs visible to clients are assigned by the inference server, not pre-mi **Staging ID**: A 128-bit random UUID minted by Charon at `POST /staging` time and returned to the proxy. It is never exposed to clients. It ties the resolved chain context to the subsequent chunk writes and commit. The proxy binds the canonical response ID to the staging ID on the first chunk write (`PUT /staging/{id}/chunks/0?response_id=...`). -**ID flow — continuation (streaming):** +**ID flow — new chain or continuation (store: true):** ``` -POST /staging?prev={prev_response_id} → Charon returns staging_id + flat_context +POST /staging[?prev={prev_response_id}] → Charon returns staging_id + flat_context inference → server assigns canonical_id PUT /staging/{staging_id}/chunks/{k}?response_id={canonical_id} (first chunk binds the ID) PUT /staging/{staging_id}/complete?response_id={canonical_id}&total={N} → committed ``` -**ID flow — continuation (buffered):** +**ID flow — store: false continuation:** ``` -GET /responses/{prev_response_id} → Charon returns flat_context -inference → server assigns canonical_id -POST /responses/{canonical_id} body: { previous_response_id, request_blob, response_blob } - → 201 Created; node committed atomically -``` - -**ID flow — new chain:** -``` -POST /responses/{canonical_id} body: { previous_response_id=null, request_blob, response_blob } - → 201 Created; node committed atomically +GET /chain/{prev_response_id} → Charon returns flat_context (no staging record) +inference → server assigns canonical_id (not stored) ``` --- @@ -242,11 +235,13 @@ The proxy is responsible for constructing the blobs sent to Charon — Charon st ## `store: false` Semantics -When the client sets `store: false`, the proxy skips the store call to Charon after inference. The response is never written to durable storage. When `store: false`, the proxy also skips the staging open call — there is no `/staging` interaction, only the prior-chain resolve (if a `previous_response_id` is present). +When the client sets `store: false`, the proxy skips all Charon store calls — no staging record is opened, no chunks are written, and no commit is issued. The response is never written to durable storage. + +For continuations with `store: false`, the proxy fetches prior context via `GET /chain/{previous_response_id}` (read-only, no staging record). For first turns with `store: false`, no Charon call is made at all. For WebSocket connections, the proxy maintains a connection-local in-memory cache of `store: false` responses so that `previous_response_id` lookups within the same connection still work. On disconnect, this cache is lost. A reconnecting client that references a `store: false` response ID receives `previous_response_not_found` and must start a new chain. -Charon has no knowledge of `store: false`. Because write-intents are created only at store time (not at resolve time), a minted response_id that never receives a store call leaves no orphaned state in Charon — there is nothing to clean up. +Charon has no knowledge of `store: false`; the proxy simply omits the store calls. --- @@ -270,13 +265,13 @@ The in-memory LRU blob cache (`chainCache`, default 64 MiB) keeps recently acces These are order-of-magnitude estimates for a single Charon instance on commodity server hardware (8–32 cores, NVMe SSD). They are not benchmarks; actual numbers depend on blob size, chain depth, and hardware. -**Write throughput — new response (`POST /responses/{id}` buffered path or `PUT /staging/{id}/complete`):** -Dominated by the Pebble write batch (WAL append + memtable insert). A single atomic commit writes one node record and one or two blob values. With Pebble's default write options, expect low thousands of commits per second at blob sizes of 10–100 KB. The staging flush path (`ResolveAndStage` + `StoreWithStaging`) issues two Pebble commits per turn — one for the staging node, one for the final node — which roughly halves peak write throughput versus the buffered path. +**Write throughput — new response (`PUT /staging/{id}/complete`):** +Dominated by the Pebble write batch (WAL append + memtable insert). A single atomic commit writes one node record and one or two blob values. With Pebble's default write options, expect low thousands of commits per second at blob sizes of 10–100 KB. The staging flush path issues two Pebble commits per turn — one for the staging node, one for the final node — which roughly halves peak write throughput versus a single-commit path. **Read throughput — `GET /responses/{id}`:** `Retrieve` issues a single `GetNode` and `GetBlobs` call. No chain walk, no LRU touch. On a warm block cache, expect sub-millisecond p50 latency and tens of thousands of reads per second for typical blob sizes. -**Chain resolution latency — `POST /staging?prev=...` or `GET /responses/{id}` (full chain):** +**Chain resolution latency — `POST /staging?prev=...` or `GET /chain/{id}`:** `walkAndTouch` calls `backend.LoadChain` (N sequential `GetNode` reads from leaf to root) followed by up to N `GetBlobs` calls for nodes not in the in-memory cache. Latency scales approximately linearly with chain depth N: - Cache-cold path: ~N Pebble key reads + ~N blob fetches. At 100 µs per Pebble read, a depth-10 chain resolves in roughly 2–5 ms; depth-100 in 20–50 ms.