refactor(knowledge): layered architecture with Service, repos, retrievers, and end-to-end suite#42
Merged
Merged
Conversation
…d search interfaces Lay down the v0.3.0 surface as additive scaffolding so existing code keeps working unchanged and downstream stages can plug in implementations incrementally. Concretely: - model.go / query.go: SourceDocument, DerivedChunk, DerivedLayer, DerivedSig, Query, Hit, Result. Layer/Mode are type aliases over the legacy ContextLayer/SearchMode so values remain interchangeable through the deprecation window. - ModeBM25 is now "bm25" (was ""); ResolveMode normalises legacy "" and the deprecated ModeSemantic to ModeBM25/ModeVector respectively. - chunking.go: UTF-8 safe ChunkText plus Chunker / ChunkSpec / NewDefaultChunker, including ChunkerOptions.Sig() for derived-data freshness checks. - repo.go: DocumentRepo, ChunkRepo, LayerRepo, ChunkQuery, LayerQuery and Candidate. Storage interfaces live on the top-level package to avoid cyclic imports with backend implementations. - searcher.go / ranker.go: SearchEngine, Retriever, Ranker, RRFRanker plus a free-function FuseHits for tooling that already has per-source hit lists. - rebuilder.go: EventKind, ChangeEvent, RebuildScope, Rebuilder. The legacy Reloader and ChangeNotifier are intentionally left untouched; they will be replaced in stage 4 once Service exists. No behavior changes for existing callers. go build / go vet / go test green across sdk and sdkx. Made-with: Cursor
Implement DocumentRepo, ChunkRepo and LayerRepo on top of any workspace.Workspace. The whole backend lives under sdk/knowledge/backend/fs and depends only on the public knowledge contract, so factories and tests can wire it without crossing into implementation internals. Highlights: - pathBuilder centralises layout (<prefix>/<dataset>/<doc> for raw content, <doc>.meta.json for Version/Metadata sidecars, .chunks.json for the dataset-level chunk index, <doc>.abstract / <doc>.overview for layer texts, <doc>.layers.json for DerivedSig). - atomicWrite stages every payload to a tmp suffix and finalises via Workspace.Rename, so partial writes are never observable. - FSDocumentRepo serialises Put per document, increments SourceDocument.Version atomically, and preserves Content verbatim (contract guarantee #4 / lossless GetDocument). - FSChunkRepo persists every chunk to .chunks.json and rebuilds an in-memory inverted index on Replace / Load. Search supports BM25, vector and hybrid recall with snapshots taken under RLock so heavy scoring does not stall writers. - FSLayerRepo stores layer text as human-readable sidecars and tracks DerivedSig per (doc, layer); Search performs layer-strict BM25 recall and pulls dataset-level layers into the result set. - CosineSimilarity is exported from the package so backends share one implementation. - 26 unit tests covering version increments, sidecar cleanup, atomic replace, persist+load roundtrip, cross-dataset search, vector and hybrid fan-out, layer-strict search, and validation guards. No behaviour changes for existing callers; legacy FSStore stays intact and is exercised unmodified by its existing tests. Made-with: Cursor
Adds knowledge/backend/retrieval implementing ChunkRepo and LayerRepo on top of any sdk/retrieval.Index. Each dataset gets its own pair of namespaces (kb_<dataset>__chunks / kb_<dataset>__layers) so backends with native namespace isolation (Postgres / SQLite) scale by dataset without scatter/gather inside one index. Notable choices: - Cross-dataset Search fans out across q.DatasetIDs (callers resolve the list via DocumentRepo.ListDatasets); empty list returns nil. - Replace prefers DeletableByFilter; falls back to List+Delete. - DeleteByDataset prefers Droppable, then DeleteByFilter, then list fallback so the same code works against memory / SQLite / Postgres. - LayerRepo enforces metadata.layer Eq filter so contract guarantee #3 (queries never cross layers) holds even on backends that ignore it. Tests use sdk/retrieval/memory.Index to cover replace, delete, fan-out, namespace isolation, layer filtering, vector mode and sig roundtrip. Made-with: Cursor
…ventReloader Adds the v0.3.0 orchestration layer on top of the repo + backend plumbing landed in stages 1-3. Old code paths (Store / FSStore / Reloader / NewSearchTool) remain untouched; deprecated.go shims and their removal are stage 5/6. New package surface (sdk/knowledge): - Service orchestrates document lifecycle + derived data + search - BM25Retriever / VectorRetriever / LayerRetriever feed SearchEngine - Query.resolvedDatasets carries the fan-out list across the engine - EventNotifier / EventReloader supersede ChangeNotifier / Reloader (rebuild scope reduced from per-doc -> per-dataset -> global) - Chunker now consumes ChunkConfig (the v0.3.0 name); the legacy ChunkerOptions struct was a dead stage-1 artefact and is gone New subpackage (sdk/knowledge/factory): - factory.NewLocal wires DocumentRepo+ChunkRepo+LayerRepo on backend/fs - factory.NewRetrieval wires repos against any retrieval.Index The factory lives in a subpackage to avoid the knowledge -> backend/fs -> knowledge import cycle. Notable design choices baked into Service: - Mode validation is centralised here (contract #2). Retrievers receive pre-resolved datasets and pre-normalised modes. - ScopeAllDatasets is resolved once via DocumentRepo.ListDatasets and pushed down via Query.withDatasets so each lane fans out identically. - Embedder doesn't expose a model identifier; ServiceOptions.EmbedSig carries that for DerivedSig freshness checks (defaults to the embedder's Go type name when unset). - LayerRetriever degrades hybrid -> bm25 when no embedder is wired, matching the chunk-tier behaviour. Tests cover Service contracts (#1..#7), retriever lane gating, RRF fusion and EventReloader debouncing/scope reduction. 39 new test cases; full sdk + sdkx suites pass. Made-with: Cursor
…eprecated shims - Add KnowledgeServiceNode + KnowledgeNodeConfig + RegisterServiceNode + KnowledgeServiceNodeSchema (knowledge/node_v2.go) backed by *Service. Surfaces "hits" ([]Hit), "by_dataset" (map) and the legacy "results" projection so existing graphs keep working. - Add NewSearchServiceTool / NewPutServiceTool (knowledge/tool_v2.go) exposing Service.Search / Service.PutDocument with the v0.3.0 schema (scope/mode/layer/threshold) and version round-trip. - Mark legacy surface as // Deprecated: removed in v0.3.0 — Store, FSStore, RetrievalStore, CachedStore, KnowledgeNode, KnowledgeConfig, KnowledgeNodeSchema, RegisterNode, NewSearchTool, NewAddTool, ChangeNotifier, Reloader, Document, SearchResult, SearchOptions, Chunk. Each deprecation cites the v0.3.0 replacement at the declaration site (staticcheck SA1019) and is indexed in the new knowledge/deprecated.go migration manifest. - KnowledgeNodeConfigFromMap honours legacy "max_layer" when "layer" is absent and accepts both float64 / int / json.Number for numeric fields. Tests: full sdk/... + sdkx/... pass. Made-with: Cursor
File-tree clean-up so v0.3.0 callers see a focused public API and
reviewers can spot the entire deprecation surface in one place.
- Move every v0.2.x symbol — Store, FSStore, RetrievalStore, CachedStore,
legacy KnowledgeNode/Config/RegisterNode/Schema, NewSearchTool/NewAddTool,
ChangeNotifier/Reloader, Document/SearchResult/SearchOptions/Chunk,
ChunkDocument/RankResults/RRFMerge/ScoreChunk/parseFrontmatter — into
a single sdk/knowledge/deprecated.go (~2.3k lines). Replacement index
lives in the file's package doc.
- Rename node_v2.{go,_test.go} → node.{go,_test.go};
tool_v2.{go,_test.go} → tool.{go,_test.go};
reloader_v2.{go,_test.go} → reloader.{go,_test.go}.
Symbol names keep the *Service* suffix until v0.3.0 (per arbitration
14=B) so callers and watcher continue to compile unchanged.
- Slim types.go to the v0.3.0 model only (ContextLayer, SearchMode,
ChunkConfig, DefaultChunkConfig, DefaultThreshold). Moved
Document/SearchResult/SearchOptions/Chunk to deprecated.go.
- Slim hybrid.go to CosineSimilarity only; RRFMerge/rrfKey moved to
deprecated.go.
- Slim search.go to textsearch aliases; RankResults/ScoreChunk/
parseFrontmatter moved to deprecated.go.
- Delete chunker.go (legacy ChunkDocument moved to deprecated.go);
chunking.go remains as the v0.3.0 chunker.
- Move ReloaderOptions onto reloader.go (shared by both EventReloader
and the deprecated Reloader).
- Move DatasetQuery into node.go (not deprecated; reused by
KnowledgeNodeConfig).
- Drop legacy *_test.go files (cached_store/embedder/fsstore/pipeline/
retrieval_store/search/search_bench/hybrid/chunker/tool) per
arbitration 13=B; v0.3.0 coverage lives in service_test, retriever_test,
reloader_test, node_test, tool_test.
Tests: full sdk/... + sdkx/... pass.
Made-with: Cursor
After the v0.2.x surface was consolidated into deprecated.go, several godoc comments still referenced "the legacy ChangeNotifier in reloader.go" and "future ChangeNotifier implementations". Rewrite them to reflect the actual layout: - ChangeNotifier / Reloader live in deprecated.go. - EventNotifier / EventReloader / Rebuilder are the v0.3.0 surface, with sdkx/knowledge/watcher as the only in-tree producer awaiting migration. Also pick up gofmt-driven alignment touch-ups in fs/document.go, fs/chunk.go, fs/layer.go and retriever_test.go that were sitting in the working tree. Made-with: Cursor
FSLayerRepo now persists embeddings in co-located binary sidecars
(<base>.vec) alongside the human-readable layer text, so the layer
tier reaches feature parity with the chunk tier on vector recall.
- paths.go: layerVecPath / datasetLayerVecPath
- vec.go: little-endian float32 codec with KVEC magic + length
- layer.go:
* Put writes/removes the sidecar in lockstep with text
* Get populates DerivedLayer.Vector from the sidecar
* Search runs BM25 and cosine lanes; ModeHybrid fuses them per
(dataset, doc) so each entity yields one Candidate
* DeleteByDoc / DeleteByDataset clean the sidecar too
- tests cover round-trip, sidecar deletion on empty re-Put,
ModeVector ranking, hybrid lane fusion and codec error paths.
Made-with: Cursor
…corpus Adds sdk/knowledge/e2e/ with a hand-curated fixture (10 clusters x 10 short markdowns) plus a 30-question golden set covering keyword direct hits, paraphrases, intra-cluster ambiguity, cross-cluster traps, and 4 negatives. Drives the canonical knowledge.Service via factory.NewLocal to catch retrieval-quality regressions that unit tests miss. Three modes are exercised through the same harness: - BM25 runs by default; recall@5 = 1.00 on the current questions - Vector and Hybrid require //go:build integration plus the standard EMBEDDING_PROVIDER / API_KEY / MODEL envs and skip cleanly otherwise - TestE2E_HybridBeatsBM25 is a relative invariant: hybrid recall must not regress more than 5% below pure bm25 on the same fixture, which is the cheapest way to catch fusion / lane-drop bugs Includes a small testenv loader copy under sdk/knowledge/e2e/internal/testenv to avoid reaching into sdkx/internal/testenv across package boundaries. Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Re-architect the
knowledgemodule from a single-package store/search blob into a layered Service + Repo + Retriever + Ranker stack, with full v0.2.x backward compatibility through a singledeprecated.goand a new end-to-end retrieval test suite. Nine commits, sequenced so every stage builds and tests green on its own.What changes
New layered architecture (commits 1–4)
sdk/knowledge): canonical types (Query,Hit,Result,DerivedChunk,DerivedLayer,Mode,Layer), repo interfaces (DocumentRepo,ChunkRepo,LayerRepo), retriever / ranker / engine contracts.backend/fs: filesystem-backedDocumentRepo,ChunkRepo(in-memory inverted index, atomicchunks.jsonper dataset),LayerRepo(BM25 + vector lanes; vectors persisted in<base>.<layer>.vecbinary sidecars).backend/retrieval:retrieval.Index-backed chunk/layer repos, one namespace per dataset (kb_<dataset>__chunks/__layers).service.go): the single orchestration entry point — validation, dataset fan-out, embedding forDerivedSig.EmbedSig, and lifecycle of derived data.factory/):NewLocalandNewRetrievalwire up the canonical Retriever set; lives in a subpackage to break the import cycle.Backward compatibility (commits 5–7)
deprecated.go, each annotated// Deprecated:and slated for removal in v0.3.0.type Layer = ContextLayer,type Mode = SearchMode) plusResolveMode("") -> ModeBM25andResolveMode(ModeSemantic) -> ModeVectorkeep legacy callers working untouched.KnowledgeServiceNode,NewSearchServiceTool,EventReloader, …) so old and new APIs co-exist during the deprecation window.Vector lane on the FS layer backend (commit 8)
backend/fs/vec.go: little-endian[]float32codec with magic header (KVEC+ length prefix), puremath.Float32bits(nounsafe).LayerRepo.Put/Get/Search/Delete*all flow vectors through new<base>.<layer>.vecsidecars;ModeHybridfuses BM25 and vector lanes by max-score per(dataset, doc).End-to-end retrieval suite (commit 9)
sdk/knowledge/e2e/: hand-written 100-doc Chinese markdown corpus (10 clusters × 10 docs) and a 40-questiongolden.jsonlcovering keyword direct, intra-cluster ambiguity, cross-cluster traps, semantic paraphrase, and 4 negatives.//go:build integrationand reuse theEMBEDDING_PROVIDER / API_KEY / MODELenvs.TestE2E_HybridBeatsBM25invariant — hybrid recall must not regress more than 5% below pure BM25 on the same fixture, which is the cheapest way to catch fusion / lane-drop bugs.Stats
go test ./sdk/knowledge/...on its ownTest plan
go vet ./sdk/knowledge/...go vet -tags=integration ./sdk/knowledge/...go test ./sdk/knowledge/... ./sdkx/knowledge/... -count=1— all greenrecall@5 = 36/36 (1.00),keyword = 36/36 (1.00)-tags=integration, real embedder):recall@5 = 36/36 (1.00),keyword = 36/36 (1.00)-tags=integration):recall@5 = 36/36 (1.00),keyword = 36/36 (1.00)TestE2E_HybridBeatsBM25(-tags=integration): hybrid 1.00 vs bm25 1.00, well within 5% toleranceMade with Cursor