feat(sdkx/retrieval/workspace): LSM-tree retrieval index on Workspace#82
Merged
Conversation
Pin the sdk dependency to v0.2.11 (CapabilityReporter) and have objstore.objectWorkspace publish its conservative envelope so adapters that rely on atomic Rename (e.g. the upcoming retrieval/workspace LSM-style Index) can detect at construction time that an object-store backed Workspace is unsuitable, instead of silently corrupting state on partial writes. objstore reports: AtomicRename: false // implemented as copy + delete ReadAfterWrite: false // conservative; concrete stores may strengthen DurableOnWrite: true // Put returns only after server commit Distributed: true // multiple processes / hosts may share a bucket Co-authored-by: Cursor <cursoragent@cursor.com>
Lay down the new retrieval.Index implementation backed by any
sdk/workspace.Workspace. This commit ships only the package shape so
the architecture can be reviewed before write-path code lands:
- doc.go: full LSM-on-Workspace architecture, recovery story,
concurrency contract, and capabilities table.
- types.go: on-disk types (manifest, segmentRef, segmentMeta,
walHeader, walRecord) plus three schema-version constants and
three sentinel errors derived from sdk/errdefs:
ErrLocked -> Conflict (another writer holds the lockfile)
ErrCorrupt -> Internal (checksum / schema mismatch)
ErrClosed -> NotAvailable (post-Close operation)
- index.go: Config + Option family (WithRoot/WithMemtableMaxDocs/
WithMemtableMaxBytes/WithWALMaxBytes/WithLockHeartbeat/WithClock/
WithAutoCompact), Index struct, namespaceState, New/Close, and a
Capabilities() that pulls AtomicRename / ReadAfterWrite /
Distributed from sdkworkspace.CapabilitiesOf so the reported
retrieval.Capabilities is accurate for whatever Workspace is
plugged in (LocalWorkspace -> WriteIsAtomic=true; objstore ->
WriteIsAtomic=false).
- errors.go: errNilWorkspace via errdefs.Validation.
WAL writer/reader, memtable, segment writer/reader, namespace
lifecycle, and the actual Upsert/Search code arrive in subsequent
commits on this branch.
Co-authored-by: Cursor <cursoragent@cursor.com>
…recovery) Implements the durable write path. Upsert/Delete append a record to the active WAL log AND stage the change in an in-memory memtable; when memtable thresholds (doc count or byte budget) are crossed, the namespace synchronously flushes a new segment under the meta-last commit protocol and atomically swaps the manifest into place. Components: - paths.go: per-namespace layout helpers (manifest, wal, segments) so all path math lives in one place. - wal.go: walWriter with rotation + size budget; walReader replays every log with seq >= manifest.FirstUnconsumedWALSeq, treats a truncated tail as EOF, and refuses on header-magic / version mismatch (ErrCorrupt). - memtable.go: insertion-order buffer with per-ID dedup; snapshot() zero-copies items out and resets so the post-flush memtable is immediately writable. - segment_writer.go: meta-last commit protocol -> docs.jsonl + docs.offsets.bin + tombstones.bin written first, meta.json written last as the segment-ready marker. CRC32 of each file recorded in meta. bm25/vector sidecars are reserved in paths.go but not yet emitted; Stage C fills them in. - namespace.go: ensureNamespace handles open-or-create with manifest read, abandoned-segment GC, WAL replay, and walWriter setup. flushLocked rotates WAL first, builds the segment, writes manifest.json.tmp, renames into place, advances FirstUnconsumedWALSeq, and best-effort retires consumed WAL files. - write.go: Index.Upsert / Index.Delete / Index.Flush. Crash recovery story: a manifest with FirstUnconsumedWALSeq advanced past a particular seq guarantees that seq's records are already in some segment, so re-replay is skipped even if the WAL file itself hasn't been GC'd. A segment whose dir exists without meta.json is an abandoned partial flush and is GC'd on next open. ErrClosed / ErrCorrupt / errEmptyNamespace surface via sdk/errdefs categories so HTTP / RPC adapters map them via errdefs.Is*. Tests cover nil workspace rejection, WAL persistence, segment production, meta-last marker, threshold auto-flush, reopen+replay, empty namespace rejection, post-Close ErrClosed (verifies errdefs.IsNotAvailable), and FirstUnconsumedWALSeq advancement across two flushes. Read path (Search/List/Get + BM25/vector) lands in the next commit. Co-authored-by: Cursor <cursoragent@cursor.com>
…arch + scoring
Implements the read side. Search/List/Get walk the manifest's
segments newest-first plus the in-memory memtable, with the rule
"freshest writer wins" for both content and tombstones.
Key design decision: BM25 corpus statistics + per-doc tokens are
NOT persisted on disk. They are pure derivatives of docs.jsonl,
so [segmentReader.loadBM25] rebuilds them in memory on first
Search by running the configured tokenizer over every doc and
folding the result into a segment-local
[textsearch.CorpusStats]. This keeps BM25 logic in one place
(sdk/textsearch is the single source of truth) at the cost of
~tens of milliseconds per cold segment load — well within the
few-thousand-doc budget the default flush threshold imposes.
A future commit can layer a non-authoritative bm25.cache.bin
sidecar on top of this if profiling justifies it.
Per-modality scoring delegates to sdk:
- BM25: textsearch.BM25(docTokens, keywords, corpus)
- cosine: scoring.CosineSim(query, doc.Vector)
- hybrid: scoring.RRF([][]Hit{bmHits, vecHits}, DefaultRRFK)
MinScore is applied only on single-modality paths; the hybrid
path returns RRF scores which live on a different scale, matching
the in-memory backend's contract.
Filter pushdown is unconditional: the workspace backend reads the
full retrieval.Doc and evaluates retrieval.DocMatchesFilter
directly, so SupportsFilter returns true for every operator.
New files:
- segment_reader.go: opens meta + tombstones eagerly, loads docs +
BM25 corpus lazily, all CRC-verified.
- read.go: Search / List / Get / SupportsFilter.
- read_test.go: BM25 ranking, vector top-k, hybrid RRF presence,
filter pushdown, MinScore, tombstone suppression across two
flushes, memtable-overrides-segment, no-query rejection,
empty namespace, list pagination, list filter, list tombstone
drop, get hit/miss/post-tombstone, CRC corruption rejection.
Modified:
- index.go: + WithTokenizer Option (default CJKTokenizer).
- segment_writer.go: tombstone-only segments now omit docs.jsonl /
docs.offsets.bin entirely; reader keys the "should I read this"
question on FileChecksums membership.
Co-authored-by: Cursor <cursoragent@cursor.com>
….Compact Adds the background compactor that merges small peer segments into larger ones, keeping per-Search reader fan-out bounded as flushes keep landing. Selection (universal-compaction shape, not strict size-tiered): sort live segments by ID, drop any over WithCompactionMaxSize, then take the OLDEST WithCompactionMinSegments contiguous candidates whose pairwise size ratio stays inside 4×. The ratio gate keeps the merge from dragging a 100 MiB segment together with a few KB of tombstones, but is loose enough that a tombstone-only segment still pairs with its immediate older sibling — exactly the case where the merge can finalise the delete. Tail-anchored merges (group includes the oldest live segment in the namespace) drop tombstones entirely: there is no still-live older segment that could resurface the deleted IDs. Non-tail merges carry tombstones forward to the new segment. Atomicity: - Source segments are immutable, so reading + writing the destination needs no namespace lock. - compactMu serialises compactions per namespace; rwMu is taken briefly only for the manifest swap. - On swap we re-read the latest manifest and remove only the exact merged refs (by ID), so a concurrent flush that landed during the merge is preserved. - Source dirs are removed AFTER the swap; a crash between swap and retire leaves orphans which the next ensureNamespace's gcAbandonedSegments cleans up. Public API: - Compact(ctx, namespace) forces one round, useful when WithAutoCompact(false) or for graceful shutdown. Background worker: - Started by New when autoCompact is true; stopped by Close via context cancel + <-compactDone wait. - Wakes on either WithCompactionInterval ticks or a non-blocking poke from flushLocked. Cap-1 wake channel so writers never block on the worker. Tests: - Min-segments threshold triggers exactly one merge. - Search result set is invariant across compaction. - Tail-anchored tombstone consolidation drops the deleted IDs permanently. - Below-threshold compact is a no-op. - WithCompactionMaxSize=1 makes every segment ineligible. - Auto-compactor converges segment count after a flush burst. - Compact on a never-touched namespace is a no-op. - Compact after Close returns ErrClosed. Co-authored-by: Cursor <cursoragent@cursor.com>
…etection
Adds a best-effort advisory locking protocol on top of Workspace
primitives (Read / Write / Rename / Delete) so two writers cannot
corrupt the same namespace by writing manifests concurrently.
Protocol:
- <ns>/.lock holds a lockState{Holder UUID, PID, Hostname,
AcquiredAt, HeartbeatAt} written via tmp+Rename.
- A second writer reads the file:
- HeartbeatAt within 3× WithLockHeartbeat: live -> ErrLocked.
- HeartbeatAt older OR file corrupt OR absent: stale -> writes
its own lockState and proceeds.
- A heartbeat goroutine refreshes HeartbeatAt every
WithLockHeartbeat. If a tick observes Holder != ourHolderID
(someone else took over), st.fenced.Store(true) and the
goroutine exits. Subsequent public methods on that namespace
return ErrFenced.
ErrFenced is errdefs.NotAvailable; ErrLocked is errdefs.Conflict
(unchanged). Both are detectable via errdefs.Is*.
Workspaces lacking AtomicRename (e.g. object stores) disable the
protocol entirely: acquireLock returns lockHolder=="" and the
heartbeat goroutine is not started. Such workspaces require
single-writer use; the package doc warns about this.
Close ordering:
1. Cancel compactor worker, wait <-compactDone.
2. Per namespace: cancel heartbeat, wait <-lockDone.
3. Close WAL writer.
4. releaseLock — but only if the file still names us as Holder,
so a fenced Index never erases a peer's lockfile.
Fence guard added to Upsert / Delete / Flush / Search / List /
Get / Compact and skipped fenced namespaces in the auto-compactor
loop, so a writer that has been taken over makes no further
changes that could conflict with the new holder.
Tests:
- Live lock rejects second acquire with errdefs.IsConflict.
- Hand-written stale lockfile is taken over silently.
- Heartbeat advances HeartbeatAt over time without changing
Holder.
- Hand-written peer takeover trips ErrFenced on the next
heartbeat tick; subsequent Upsert returns errdefs.IsNotAvailable.
- Fenced Close does not delete the peer's lockfile.
- Clean Close removes our own lockfile.
Co-authored-by: Cursor <cursoragent@cursor.com>
…ce + DeleteByFilter Adds the standard contract.Run() suite as a sub-test, parameterised by a fresh-MemWorkspace factory. The workspace backend now passes every applicable test in the same suite that sdk/retrieval/memory and sdk/retrieval/postgres satisfy: ✓ UpsertGetDelete ✓ FilterEqAndIn ✓ UpsertIdempotent ✓ FilterRangeAndExists ✓ ReadAfterWrite ✓ FilterNotComposes ✓ NamespaceIsolation ✓ DeleteByFilterValidation ✓ SearchNoQuery ✓ CapabilitiesShape ✓ ListPagination ⤬ Skip: SearchDebugIncludeLanes ✓ ListPaginationStalePageToken (Capabilities.Debug=false by design) Two contract-driven changes: 1. Removed the search-side hasContribution gate that suppressed zero-score docs. The contract (matched by memory backend) is "every filter-passing doc is a candidate; the caller decides ranking/cutoff via MinScore + TopK". Without this fix the degenerate "QueryText='x' (one-char, tokenizes to nothing) + Filter only" pattern that FilterRangeAndExists exercises returned no hits. 2. Implemented retrieval.DeletableByFilter, which the Capabilities.NativeDeleteByFilter field had been advertising without a concrete method. Empty filters reject with retrieval.ErrEmptyDeleteFilter (errdefs.Validation), matching memory backend. Internally List + Delete to keep the WAL / segment story uniform; cost is one tombstone per matched doc, which compaction folds away on the next round. Drive-by: fixedClock test helper now uses atomic.Int64 so TestCompact_AutoTriggeredByFlush is race-free with the background compactor goroutine. Updated the BM25 ranking test to assert "charlie ranks below alpha/bravo" instead of "charlie absent", since the contract now requires zero-score docs to be returned. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A new
retrieval.Indexbackend that persists docs/vectors on anysdk/workspace.Workspacevia an LSM-tree (WAL + memtable + immutablesegments + compaction). Lets MemWorkspace, LocalWorkspace, and the
objstore Workspace all serve as a retrieval store with no extra deps.
Key design decisions:
Segments only persist
docs.jsonl; readers rebuildtextsearch.CorpusStatson first BM25 query and delegate scoringto
sdk/textsearch+sdk/retrieval/scoring(full reuse, noduplicated formulas).
size-ratio constraint; tail-anchored merges drop tombstones.
<ns>/.lockwith randomHolder UUID and heartbeat. Stale (3× heartbeat) → takeover.
Lost lock → namespace fenced, all subsequent ops return
ErrFenced. Disabled when Workspace lacksAtomicRename.sdk/errdefs—ErrLocked(Conflict),ErrCorrupt(Internal),ErrClosed/ErrFenced(NotAvailable).Stages, one commit each:
Index.Compactsdk/retrieval/contractconformance +DeleteByFilterPlus a small
sdkx/workspace/objstorechange to advertise itscapabilities.
E2E tests against a real on-disk LocalWorkspace land in a follow-up
PR (own module, build-tagged, version-pinned) once this PR's sdkx
tag is cut.
Test plan
go test ./retrieval/workspace/... -count=1go test ./retrieval/workspace/... -race -count=1sdk/retrieval/contractsuite green viacontract_test.gouniversal compaction, lock takeover, fence detection,
DeleteByFilter
Made with Cursor