v0.3.0 — Object-Storage Persistence, Typed Relations, Deletes
Compass 0.3.0 turns the embedded engine into a cloud-durable one: your own object storage can now be the source of truth, while local-first stays the zero-config default.
Highlights
🪣 Object-storage persistence (opt-in) — build with --features object-storage and set COMPASS_STORAGE=s3://bucket[/prefix] (or gs://, az://). Every write lands durably in your bucket first — an LSM of immutable WAL fragments committed through a CAS'd manifest, with automatic compaction and physical GC. Destroy the container and its disk; on the next boot Compass discovers every collection in the bucket and rebuilds all local indexes — chunks, hierarchy, and relations included. Works with AWS S3, GCS, Azure Blob, MinIO, and R2; an optional key prefix lets multiple deployments share one bucket.
🔗 Typed chunk relations — directed, labeled, many-to-many edges between chunks (cites, supersedes, anything), disk-backed in redb with multimap endpoint indexes. Create in batches, query per chunk with direction/type filters, or return them inline with search hits. Dangling targets are reported (target_status: "missing"), never silent.
🗑️ Deletes + compaction — soft-delete by id or metadata filter (DELETE /chunks/:id, POST /delete); tombstoned chunks vanish from results immediately and are physically reclaimed by compaction (POST /compact, or automatic in cloud mode).
🔢 True u64 chunk ids — the filter index moved from RoaringBitmap to RoaringTreemap, and the mmap vector file gained a u64-count v2 header (legacy files read transparently and migrate on first append). No more silent truncation past 4.29B ids.
Hardening
This release went through three adversarial review rounds. Among the bugs found and regression-tested before release: concurrent-append data loss, local↔cloud split-brain in both directions, a compaction path that leaked objects unboundedly, chunk-id reuse after compaction, GCS conditional writes requiring the object generation rather than the ETag, silently-dropped bucket prefixes, torn mmap appends, and missing embedding-dims validation. Details in the CHANGELOG.
Testing
99 tests in the default build, 125 with object-storage — including fault injection, 16-writer concurrent appends, torn-append recovery, and restart recovery on both fresh and persistent disks. Env-gated integration tests run the CAS and LSM lifecycle against a real S3 endpoint (verified on MinIO): see crates/compass/src/storage/s3_integration_tests.rs for the recipe, or docker compose -f docker-compose.minio.yml up to try it locally.
Scope, honestly
Cloud mode is durable via object storage, fast via local: reads serve from locally rebuilt indexes, so this is not stateless multi-node serving yet, and cold-start recovery materializes a collection's live set in RAM. Those are the next frontier — see the CHANGELOG's scope notes.
Full changelog: v0.2.0...v0.3.0