feat(core): SPI split — RetrievalBackend / IndexBackend (Step 1.1b)#45
Merged
officialCodeWork merged 1 commit intoMay 24, 2026
Merged
Conversation
Split VectorStore / KeywordStore / GraphStore into narrow read+write roles (*RetrievalBackend / *IndexBackend) so consumers depend on the role they actually use; legacy composite ABCs preserved for backends serving both sides. Adds bulk + streaming + ID-only retrieval to the SPI surface, introduces IndexHint + WriteVolume for scale-tier writes (ADR-0009), and splits Embedder into single embed + canonical bulk_embed. - Retrieval side: retrieve_ids → list[ChunkRef]; hydrate on keyword stores loads full Chunks (vector stores pass-through, content lives elsewhere). - Index side: bulk_index / bulk_delete; stream_index default batches an AsyncIterator into bulk_index calls. Graph adds bulk_upsert_nodes/edges and bulk_delete_nodes + stream_upsert_nodes. - Backends migrated: NoopVectorStore / NoopKeywordStore / NoopGraphStore / NoopEmbedder + PgVectorStore + QdrantVectorStore. - tests/contract/spi_signature.py extended to enforce ctx-first on all new role ABCs. - Schemas regenerated (IndexHint.json now produced by gen_schemas). - Docs: rag-core.md adds "Read/write SPI split" + IndexHint section; storage-backends.md points to the new method names. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
officialCodeWork
pushed a commit
that referenced
this pull request
May 24, 2026
… (Step 1.1c)
New workspace package packages/policy/ (import root rag_policy v0.1.0).
Establishes the single Policy Decision Point ADR-0005 calls for: every
retrieval / ingest / egress code path consults PolicyEngine, and a CI
linter fails when a governance-relevant SPI call lands without an
adjacent consultation.
Surface
- PolicyEngine ABC: async evaluate(ctx, decision, subject) -> PolicyResult
and async filter_pushdown(ctx, decision) -> FilterExpr.
- NoopPolicyEngine: always-ALLOW; filter_pushdown still emits a
tenant-scoped And(Eq("tenant_id", ...)) so backends never cross-tenant
leak even with the noop loaded.
- PolicyWriter facade (mirrors AuditWriter): delegates to engine, emits
policy.decision structured log via rag-observability.
- PolicyDecision enum: read_chunk / ingest_doc / egress_text /
quota_check / rate_limit / execute_plan.
- PolicyResult: frozen union (allow / deny(reason) / transform(subject))
with is_allow / is_deny / is_transform / transformed_or helpers.
- QuotaSubject / RateLimitSubject for the non-Chunk decision subjects.
- FilterExpr mini-language (Eq, AnyIn, And, Or, Not, TrueExpr):
discriminated-union of frozen Pydantic models; backends translate to
native filter languages.
Coverage linter
- tests/policy/coverage.py greps for retrieve_ids / hydrate / bulk_index
/ stream_index / bulk_embed / .complete call sites without an adjacent
PolicyEngine / PolicyWriter marker. File-allowlist at top; failures
block CI. Consumers (gateway, ingest) shrink the allowlist as they
wire the PDP in. Future Step 1.1f tightens to call-pattern matching.
- Collected via pytest python_files extended to include coverage.py.
Wiring
- packages/policy added to [tool.uv.workspace].members and to pytest
pythonpath in root pyproject.toml.
- Dependencies: rag-core + rag-observability only (matches backends
precedent; CLAUDE.md graph: policy -> core).
Tests + gates
- 20 conformance tests under tests/contract/test_policy_engine.py.
- 1 coverage-linter run under tests/policy/coverage.py.
- ruff + mypy --strict (83 source files, +7 from 1.1b) + RAG001 logging
check all green; full non-integration suite 528 tests pass.
Docs
- New docs/reference/rag-policy.md (Overview / Usage / Internals /
Extension points), docs/README.md index updated. ADR-0005 and
docs/architecture/policy-engine.md already match the shipped shape.
TRACKER housekeeping
- Step 1.1c flipped to done; Next action set to 1.1d (Pipeline +
Batcher primitives).
- PR history table: added rows for #44 (1.1a) and #45 (1.1b) which
were missed when those steps merged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
officialCodeWork
pushed a commit
that referenced
this pull request
May 24, 2026
… (Step 1.1c)
New workspace package packages/policy/ (import root rag_policy v0.1.0).
Establishes the single Policy Decision Point ADR-0005 calls for: every
retrieval / ingest / egress code path consults PolicyEngine, and a CI
linter fails when a governance-relevant SPI call lands without an
adjacent consultation.
Surface
- PolicyEngine ABC: async evaluate(ctx, decision, subject) -> PolicyResult
and async filter_pushdown(ctx, decision) -> FilterExpr.
- NoopPolicyEngine: always-ALLOW; filter_pushdown still emits a
tenant-scoped And(Eq("tenant_id", ...)) so backends never cross-tenant
leak even with the noop loaded.
- PolicyWriter facade (mirrors AuditWriter): delegates to engine, emits
policy.decision structured log via rag-observability.
- PolicyDecision enum: read_chunk / ingest_doc / egress_text /
quota_check / rate_limit / execute_plan.
- PolicyResult: frozen union (allow / deny(reason) / transform(subject))
with is_allow / is_deny / is_transform / transformed_or helpers.
- QuotaSubject / RateLimitSubject for the non-Chunk decision subjects.
- FilterExpr mini-language (Eq, AnyIn, And, Or, Not, TrueExpr):
discriminated-union of frozen Pydantic models; backends translate to
native filter languages.
Coverage linter
- tests/policy/coverage.py greps for retrieve_ids / hydrate / bulk_index
/ stream_index / bulk_embed / .complete call sites without an adjacent
PolicyEngine / PolicyWriter marker. File-allowlist at top; failures
block CI. Consumers (gateway, ingest) shrink the allowlist as they
wire the PDP in. Future Step 1.1f tightens to call-pattern matching.
- Collected via pytest python_files extended to include coverage.py.
Wiring
- packages/policy added to [tool.uv.workspace].members and to pytest
pythonpath in root pyproject.toml.
- Dependencies: rag-core + rag-observability only (matches backends
precedent; CLAUDE.md graph: policy -> core).
Tests + gates
- 20 conformance tests under tests/contract/test_policy_engine.py.
- 1 coverage-linter run under tests/policy/coverage.py.
- ruff + mypy --strict (83 source files, +7 from 1.1b) + RAG001 logging
check all green; full non-integration suite 528 tests pass.
Docs
- New docs/reference/rag-policy.md (Overview / Usage / Internals /
Extension points), docs/README.md index updated. ADR-0005 and
docs/architecture/policy-engine.md already match the shipped shape.
TRACKER housekeeping
- Step 1.1c flipped to done; Next action set to 1.1d (Pipeline +
Batcher primitives).
- PR history table: added rows for #44 (1.1a) and #45 (1.1b) which
were missed when those steps merged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Step 1.1b of the Phase 1 architecture-refactor window. Splits the storage-side
SPIs into narrow read / write roles, formalises ID-only retrieval, and
introduces the
IndexHintparameter that ADR-0009 calls for.VectorStore/KeywordStore/GraphStorebecome thincomposite ABCs over
*RetrievalBackend(read) and*IndexBackend(write).Consumers can depend on the narrower role; backends serving both sides
keep one class to implement.
retrieve_ids(ctx, …) → list[ChunkRef]replaces theold
query/searchmethods that returned(ChunkId, score)tuples.hydrate(ctx, chunk_refs)loads fullChunkpayloads from keyword stores;vector stores pass-through (content lives elsewhere — Storage SPI / keyword
index).
bulk_index/bulk_deleteare the canonicalwrite methods.
stream_indexaccepts anAsyncIteratorand batches intobulk_indexcalls (default impl; backends with native streaming canoverride). Graph adds
bulk_upsert_nodes,bulk_upsert_edges,bulk_delete_nodes,stream_upsert_nodes.IndexHint+WriteVolumeadded torag_core.types(ADR-0009);passed to all
bulk_index/stream_indexcalls and reserved forindex-implementation selection at
initialize()time.bulk_embed(ctx, texts, chunk_ids); thinembed(ctx, text, chunk_id)default wrapsbulk_embedfor single-itemcallers (per performance.md p99 budgets).
tests/contract/spi_signature.py) extended toenforce ctx-first on the six new role ABCs.
NoopVectorStore,NoopKeywordStore,NoopEmbedder,PgVectorStore,QdrantVectorStore(NoopGraphStore inherits the newbulk defaults unchanged).
Embeddingstill lacks a typedcorpus_idfield (ADR-0004 gap); pgvectorand qdrant continue to default to
""for now — addressed in a later step.Documentation
SPI split (Step 1.1b)" and "Choosing an
IndexHint" sections; updates thetype table with
IndexHint/WriteVolume.— points the "add a new VectorStore backend" checklist at the new method
names.
IndexHintshape and per-size index choice; this PR is the SPI implementation it
describes.
Test plan
uv run ruff check .+uv run ruff format --check .— cleanuv run mypy packages/ apps/gateway/— 76 source files, no issuesuv run python scripts/check_logging.py— RAG001 cleanuv run pytest tests/ packages/ --ignore=tests/integration -x -q— 507 passed, 1 skippeduv run python -m rag_core.gen_schemas dist/schemas—IndexHint.jsongeneratedtests/integration/test_pgvector.py,test_qdrant.py)updated for the new method names — run with
task devservices upbefore merge.
🤖 Generated with Claude Code