Add a durable backend: serverless, cross-machine memory on object storage

[ShawnChen-Sirius]

What

Adds a third memory backend, durable: the clickmem database lives in object storage you own (S3 / GCS / Azure Blob / any S3-compatible / a local folder), so memory persists and moves across machines without running a server. Built on chdb.durable (chdb-io/chdb#616).

Proposal and discussion: #6

Why

Fills the gap between local (persistent but one machine) and clickhouse (portable but needs a server): durable, portable, and serverless — for agents that restart on a fresh host (Lambda, CI, sandboxes).

Kept true to clickmem

• A memory write is saved before it returns; recall reads a local copy (no network). Background bookkeeping (events, raw transcripts) is uploaded lazily so it never slows the hot path; a crash can lose only recent bookkeeping, never a memory.
• Storage can't grow without bound — the change log is periodically compacted back into a fresh snapshot.
• Choosing a store is deliberate: the active store name is saved per user in ~/.clickmem/stores.json (not in the repo). New store list|use|new (CLI) and clickmem_store (MCP), shaped like the existing project / blacklist commands.

Compatibility

Uses chdb.durable, which is merged into chDB but not yet in a published release, so this backend stays inactive until a chdb release includes it. The local and clickhouse backends are unchanged.

Tests

151 pass (+5 for store selection). The durable backend has a standalone scenario script — bootstrap, persist, restore, log replay, vector search, crash recovery with the single-writer lock, lazy bookkeeping with bounded loss, and storage compaction — verified locally and against MinIO.

Design document

Full design (click to expand)

ClickMem Durable Memory Backend (on chdb.durable)

1. Goal

ClickMem memories currently live in one of two places: the local disk (the local backend — gone when you switch machines), or a ClickHouse server (the clickhouse backend — someone has to run a server). This adds a third option: memories persist in object storage the user owns (S3 / GCS / Azure Blob / a local folder). No server, and the same memory works across machines, clouds, and throwaway environments like Lambda.

Backend	Lives in	Durable	Cross-machine	Needs a server	Concurrent writes
local	local disk	yes	no	no	single process
durable (new)	your object storage	yes	yes	no	single writer
clickhouse	ClickHouse server	yes	yes	yes	multiple writers

2. What chdb.durable is and how it works

chdb.durable is a subpackage merged into the chDB main repo (chdb-io/chdb PR #616, merged 2026-07-23). It provides a persistent, addressable embedded analytical object; clickmem is one consumer of it.

Storage layout — one memory store = one directory in object storage:

s3://my-bucket/clickmem/<store-id>/
├── head.json      metadata: who is using it (lease) + which files hold the data (manifest)
├── base           full snapshot: one complete backup of the chDB database
└── wal-*.jsonl    incremental log: every write SQL since the snapshot, in order

Three core flows:

• Open ns.open(id): read head.json. Missing → create a fresh empty store. Present → download the base, restore it into a local chDB database, then replay the incremental log in order — state is back to where it was. All queries afterwards hit this local replica; no network involved.
• Write execute + flush: the write runs locally and its SQL is buffered; flush packs the buffer into one log file, uploads it, and updates head.json. When flush returns, the data is in object storage — a crash right after loses nothing.
• Checkpoint: fold "snapshot + a chain of log files" into one fresh snapshot and clear the log. It runs on close or when thresholds trip; it is an optimization, not a requirement — skipping it only makes the next open replay a longer log.

Preventing two processes from corrupting the same store (single-writer lease): head.json records the current holder and an expiry (60s by default; an active writer renews automatically). Every head.json update uses the object store's conditional write — the overwrite only succeeds if the file's version hasn't changed since it was read. So even if a stale process is still alive, once someone else takes over, the stale process's next commit is rejected by the storage layer itself. Crash recovery: wait out the 60s lease, or pass force=True once you are sure the old process is dead.

Two rules for statements in the incremental log (violating either corrupts crash recovery):

1. Write statements must be deterministic — generate timestamps and ids in the caller and put them in the SQL as literals. Never use now() / rand(): replay would evaluate them again and produce different values.
2. Unqualified table names replay with the store's own database as the current database (chdb.durable guarantees this). Statements touching any other database must name it explicitly.

Growth control

Logs and files must not grow without bound. Four gates, one principle: maintenance always rides normal operations — no dedicated timer thread (the lease already renews the same way, inside writes):

Prevents	Gate	Checked
Too many log segments (long-running service, ever-longer replay chain)	≥ 64 segments → fold into a fresh snapshot	after every belief write
Log too large (transcript-heavy sessions: few segments, many bytes)	≥ 32 MB written → fold	same
Deferred audit writes piling up in bytes	≥ 1 MB → upload	on audit writes
Deferred audit writes piling up in time (quiet session; a crash would lose too much)	≥ 60 s → upload	on every query, plus the server's background loop (wakes ≤ every 5 s) as a backstop

Close is graded: folding is only worth a full-snapshot upload if the log passed 8 segments; below that, just upload the remaining log — this also stops minting an orphaned snapshot on every service restart.

Not done yet: garbage collection. Superseded snapshots and already-folded log segments stay in object storage (a known chdb.durable TODO). Bounded impact: open only downloads the files head.json references, so garbage never affects speed or correctness — it only costs storage (snapshots are hundreds of KB; ~tens of MB per year). A real fix needs a "list files" primitive in the storage abstraction plus delayed deletion after folding — tracked as a chdb.durable follow-up.

3. Architecture

   Claude Code / Cursor / ChatGPT / scripts   (hooks · MCP · REST · CLI)
                             │
   ┌─────────────────────────▼─────────────────────────┐
   │            clickmem  (memory semantics)            │
   │   remember / edit / forget / pin / blacklist       │
   │   recall (runs on the local replica, no network)   │
   │   conflicts / resolve                              │
   │   ★ new: store list / store use / store new (§6)   │
   ├────────────────────────────────────────────────────┤
   │    Backend protocol (query / execute / vector_search)
   └────┬───────────────────┬───────────────────────┬───┘
        │                   │                       │
  LocalBackend       ★DurableBackend          ClickHouseBackend
  local disk         chdb.durable object      ClickHouse server
                            │
                            │  download/upload + conditional writes
                            ▼
              s3://bucket/clickmem/
              ├── _catalog.json   directory: id ↔ description map (§5)
              └── <store-id>/  head.json · base · wal-*

Division of labor: clickmem decides what counts as memory, how recall works, and how conflicts are resolved; chdb.durable decides how state survives process death; the object store's IAM decides who can touch which bucket. chdb.durable does no cataloging, no auth, no memory semantics.

4. Store ids: recorded per user locally; the directory is for exceptions

Everyday path (no directory involved): the store id is recorded in the user's own ~/.clickmem/ config (the active id, plus an optional project → id map). Every start reads the local record and opens directly. Nothing is written into the repository or git — different people can maintain different memories for the same project under their own ids. A team that wants one shared id has each member run store use <id> once, or distributes the existing CLICKMEM_DURABLE_ID environment variable. This matches clickmem's existing pattern: project_id is likewise derived at runtime from the git remote; no config file ever lands in the repo.

Creating (store new): explicitly creates a fresh empty store and switches to it. Default name = a readable project/user slug, sanitized to the id character rules — ids may not contain / \ . .. (so one id's storage prefix can never contain another's); acme/foo becomes acme-foo. If the directory already has that name, prompt to reuse or rename; --id overrides entirely. Readable names, no random hashes — users must recognize a store in a list.

Switching machines / stores (store list + store use): on a new machine, after losing the local record, or to switch stores, store list reads the directory and shows "id ↔ description ↔ source repo ↔ last used"; the user picks, store use <id> records the choice locally.

Why no automatic selection: auto-routing to the wrong store injects one project's memories into another's context — exactly the contamination clickmem exists to prevent. The system's job is to provide enough information (local record + directory); the choice belongs to the user. Same principle as clickmem's write side: memory is never created by accident, and never loaded by accident.

Store id and project_id are different layers: project_id (e.g. auxten/clickmem) is a column inside the memories table — a recall filter; the store id says which database (all tables) lives where. One store holds any number of projects. Do not create one store per project — cross-project recall only works inside a single store, and global memories would have nowhere to live. Recommended: one store per person (or per agent).

5. The directory: _catalog.json

One small JSON file per storage location: [{id, description, source, last_opened, count}]. It only serves the exception path in §4.

• Read: always one download, regardless of store count (a thousand stores ≈ 200 KB).
• Write: each process updates its own store's entry on open/close. Concurrency is safe twice over: a store's entry is only written by that store's lease holder (already serialized); updates to different entries use conditional writes — read the latest version, change only your own row, overwrite only if the version hasn't moved, retry on a fresh copy if it has. Retries always merge cleanly because writers touch disjoint rows.
• Degraded mode: the directory is a navigation hint; the truth is each store's head.json. A failed directory update can simply be dropped — at worst "last used" shows stale.
• Descriptions: derived mechanically from data (recent commit kind/tag counts, pinned titles, totals) or set explicitly by the user. Never LLM-generated — clickmem runs no LLM anywhere, metadata included.

6. API additions: one store group, named after existing conventions

Naming follows clickmem's own patterns — CLI is "noun + short verb" (precedents: project link, service install, dashboard open); MCP is one tool per noun with an op parameter (precedents: clickmem_project(op=link/list), clickmem_blacklist(op=add/remove/list)):

• CLI: store list / store use <id> / store new [--id NAME]
• MCP: clickmem_store(op="list"|"use"|"new", id=...)

No durable prefix, no durable special-casing — "list my memory stores, switch, create" makes sense for all three backends:

Backend	list	use	new
local	list databases under the local data dir	reopen at another path	create a directory
durable	read _catalog.json	close current (release) → open target (restore)	cold-create + register in the directory
clickhouse	list memory databases on the server	switch database	create a database

Every existing API (remember/recall/…) is untouched — they depend only on the Backend protocol, so the backend swap is transparent.

7. Teams and multi-tenancy

• Org-wide sharing: a shared bucket; each member points at the same store id. Reads are unrestricted (read-only opens take no lease); writes are single-writer — for standing multi-writer teams, use the clickhouse backend rather than bending this one.
• Team memory works best as "one store per person plus one team store, with explicit promotion into the team store" — consistent with clickmem's rule that memory is committed deliberately.
• Tenant isolation is the object store's IAM: same bucket with separate stores (fine internally) or separate buckets and credentials (across companies). The library treats both identically.

8. Measured performance (2026-07-27, chdb 4.2.1, 256-dim embedding workload)

Operation	Local part	In-region S3 end-to-end
single write (execute)	1.2 ms	same
flush (upload)	0.6 ms	~40–80 ms (two uploads)
checkpoint (100k memories)	13 ms, snapshot 266 KB	~60–100 ms
reopen + restore (100k)	28 ms	cold open < 0.5 s overall (measured: AWS 0.46 s / GCP 0.60 s at 1M rows)

Takeaway: recall has zero network cost; writes are infrequent and deliberate, and tens of milliseconds disappear inside an LLM turn; checkpoint needs no async machinery (durability comes from flush — checkpoint is just folding).

[ShawnChen-Sirius]

@auxten please review this PR and trigger the ci workflow.