v0.4.0 — Serverless: stateless writers, tenant partitions, serve-from-storage

Compass v0.4.0 — Serverless: stateless writers, tenant partitions, serve-from-storage

v0.3.0 made object storage the source of truth. v0.4.0 makes Compass a
serverless database: nodes are disposable, tenants are cheap, and a cold
collection answers its first query in milliseconds — while local-first,
zero-dependency embedded mode remains byte-for-byte the default.

Warm serverless (roadmap Phases 0–3)

• Stateless writer role (COMPASS_ROLE=writer): append-only nodes with no
  local indexes and instant boot. Durable immediately; searchable on serving
  nodes within the refresh interval.
• CAS id-block allocator: attached nodes and writers can never mint
  colliding chunk ids.
• Bucket collection config: vector-space specs survive cold rebuilds.
• Manifest refresher + read-your-writes: serving nodes converge on other
  nodes' writes (default 5s); writes return seq; min_seq on search gives
  read-your-writes with a bounded wait.
• Lazy attach + LRU detach (COMPASS_LAZY_ATTACH, COMPASS_MAX_ATTACHED).

Tenant partitions (Phase 6)

• config.partition_by = "tenant_id" at create: every chunk routes to an
  internal per-tenant partition — a full engine namespace behind one
  collection API. Searches/deletes filter by the partition field (set
  membership fans out ≤16, merged); ids stay collection-unique; partitions
  auto-create on first ingest (writer role included), attach on demand, hide
  from listings, cascade-delete with the parent.
• Measured: 50–200 tenants in one collection, boot 0.6s / ~15MiB RSS
  regardless of tenant count; serving RAM tracks the hot-tenant set.
• Works fully in LOCAL mode too (partition dirs on disk, shared id file).

Serve-from-storage (Phase 5) — true serverless

• COMPASS_COLD_SERVE=true: semantic queries on UNATTACHED collections and
  partitions are answered straight from object storage — manifest read,
  cached centroids, a few cluster range-GETs, byte-range hydration.
• Segment format CSEG0003: IVF-clustered vectors + row-addressable metadata,
  built at compaction. v2 segments stay readable; pre-v0.4 readers fail loudly
  on v3 (do not mix writer versions against one bucket).
• Measured (300k × 64d, MinIO): fresh node boots 0.3s at 28MiB; FIRST query
  70ms; steady state p50 25ms / p95 37ms at 66MiB RSS — vs minutes and ~2GiB
  for the attach path.
• Cold reads see the full committed state including the WAL tail —
  read-your-writes by construction. Metadata filters apply. FTS on a cold
  namespace returns a clear error (attach still builds full local indexes).
• COMPASS_WARM_AFTER (default 3) promotes hot namespaces to a background
  attach: cold → warm → hot automatically.

Fixed

• Facets: wiped by every ingest after the first, empty after restart, counted
  deleted chunks (latent since v0.2) — now chunk-id-keyed roaring treemaps.
• Warm restarts were O(collection) (full HNSW rebuild on a legitimately-stale
  index file): now incremental heal; 100k restart 20.2s → 1.6s.
• Vector-space rebuild completion never activated the space or hot-loaded the
  index until restart.
• Missing collections returned HTTP 500/400; typed not-found now maps to 404.
• Sub-1000-vector keymap persistence (wrong ids under non-dense allocation).

Lean pass

• ~1,200 lines of dead weight removed (unwired GPU backend plumbing, dead
  filter evaluators, never-wired persistence codecs, rayon); dead-code lint
  re-enabled crate-wide; one filter semantics (roaring pushdown) for search
  AND delete-by-filter; collections/mod.rs halved.

Compatibility & migration

• Local mode: no changes required; no cloud features activate without cloud
  config (guard-tested).
• Cloud mode: pre-v0.4 namespaces migrate organically (bucket config
  back-filled, id allocator seeded from the high-water mark). Do NOT run
  v0.3 and v0.4 writers against the same bucket during a rolling upgrade;
  v0.3 readers cannot read v0.4 (CSEG0003) segments and fail loudly.
• New envs: COMPASS_ROLE, COMPASS_REFRESH_INTERVAL, COMPASS_LAZY_ATTACH,
  COMPASS_MAX_ATTACHED, COMPASS_COLD_SERVE, COMPASS_WARM_AFTER,
  COMPASS_COLD_NPROBE, COMPASS_MAX_CONCURRENCY (all documented in
  .env.example; all optional).

Evidence

• 106 local / 159 object-storage tests; clippy clean under -D warnings (lint
  fully enabled); four build combinations verified in CI (including the
  non-test cloud build that only Docker used to compile).
• 58-check live E2E across full node + writer node + cold node on MinIO.
• Live benches: v0.3.0 comparison (+48% ingest, −57% RSS, −72% bucket bytes,
  equal search), 200-tenant bounded-RAM proof, 300k cold-serve proof.