rfc: search contracts and retrieval algebra - #606
Conversation
The public proposal required by the governance reset of PR #595 (size-L): retire the provably inert fuzzy() with a stable T25 diagnostic, reject silently-dropped search/rank targets on traversal-introduced bindings (T26), state retrieval in the lowered plan instead of re-inferring it from order_by[0], make rank metrics projectable with structural identity to the executed retrieval, give every ranked result a total deterministic order (fusion and aggregation included), and add the additive warnings/metrics/retrievals read envelope with exact prefiltered embedding coverage. The RFC follows the template's section set and additionally carries a clearly-marked non-normative agent-context appendix: substrate dependencies and their validated assumptions, cross-PR composition contracts (#587 gate, RFC 0040/0043 adjacency), and the build/CI traps observed while producing the prototype evidence (closed PR #595, branch retained). Registry: adds the 0047 row and advances next-available to 0048 (the stale 0045 reservation clause is retired — 0045 has landed).
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
The decision log and agent appendix referenced an internal design-graph record by identifier. Public history carries the public facts only (the prototype PR, the evidence classes); the larger program is described without internal addressing, per the change-discipline rule against private planning shorthand in public history.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b7ce5a344
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - **Physical acceleration is derived (7):** preserved — the unindexed-column | ||
| condition warns, it does not fail; index absence changes cost and (on the | ||
| flat fallback) analysis behavior, which is exactly what the warning makes | ||
| visible. Recall reporting is contractual, not plan-derived. |
There was a problem hiding this comment.
When the requested column has no FTS index, this design explicitly permits a case-sensitive fallback whose results differ from indexed execution and only adds a warning. The same query can therefore gain or lose rows after index reconciliation even though an index is derived state; making that discrepancy visible does not restore logical correctness. Use analyzer-equivalent fallback behavior or fail closed instead.
AGENTS.md reference: AGENTS.md:L99-L100
Useful? React with 👍 / 👎.
| - **Coverage.** Ready/pending counts reuse the scan's own structured | ||
| predicate through a sealed, streaming count on the storage boundary — no | ||
| SQL strings, no retained batches, computed only for `@embed`-backed vector | ||
| retrievals. |
There was a problem hiding this comment.
For an indexed nearest query with a small limit and no selective filter, computing exact ready/pending counts through a streaming predicate still scans the entire source/vector population on every request. Streaming bounds retained memory, but not the O(table-size) I/O and latency added to an otherwise sublinear ANN read; coverage must use bounded metadata, be opt-in, or otherwise avoid an unconditional full-population count.
AGENTS.md reference: AGENTS.md:L104-L105
Useful? React with 👍 / 👎.
| 1. `fuzzy()` is retired with a stable `T25` compile diagnostic — it provably | ||
| never matched under the supported tokenizer, so every use was a confident | ||
| empty answer. |
There was a problem hiding this comment.
The cited characterization in crates/omnigraph/tests/search.rs:1764-1787 exercises only one capitalized, stem-sensitive typo (Introductio), which cannot establish that every fuzzy() invocation returns empty. Fuzzy matching includes zero-edit matches, so normalized terms such as lower-case deep can still match indexed terms; rejecting every use with T25 would break working stored queries despite the compatibility section calling all affected usage provably broken. Add representative exact/stem/case/max-edit evidence or narrow the retirement to the actually inert shape.
AGENTS.md reference: AGENTS.md:L151-L153
Useful? React with 👍 / 👎.
| A complete prototype exists (closed PR #595, branch | ||
| `search-contracts-p0-p1`, retained as evidence per the closure note): eleven | ||
| staged commits, canonical workspace graph green (2,860 tests), both Clippy | ||
| gates, OpenAPI regenerated, vocabulary-guard inventory classified. Test | ||
| owners extended, not forked: compiler typecheck/lowering suites (T25, T26, |
There was a problem hiding this comment.
This Lance-dependent RFC reports only that an unspecified impact analysis and source validation occurred; it does not identify the complete upstream index, FTS, tokenizer, vector, or DataFusion pages reviewed. The RFC process requires the exact version and surveyed surfaces so acceptance can verify that the fallback, tie, and index-lifecycle assumptions were checked against the pinned substrate rather than an unavailable prototype branch.
AGENTS.md reference: AGENTS.md:L13-L17
Useful? React with 👍 / 👎.
The broader search-contracts program as a public draft RFC: three composable search contracts, schema-owned analyzed semantics (@analyzed with substrate-identity fingerprints), schema-bound vector distance and mandatory embedding-space identity, match_terms/knn/ann with a typed oversample dial, N-arm weighted rrf, typed metric domains, and the SchemaIR/format rebuild boundary they require. Blocked on RFC 0047 (its first slice); cross-links added in 0047 and the registry (next available number is now 0049).
ragnorc
changed the title
ragnorc
changed the title
0047 and 0048 are allocated by PR #606, so the RFC becomes 0049 and the registry's next number 0050. From review: the ledger restore seam is withdrawn, since its real use is a coherent restore point where the ledger and the graphs come back together; readiness reports counts only and the graph ids stay behind the authenticated GET /graphs; the shutdown watchdog is an operating-system thread armed by a listener installed before graphs open; the authority enum gains `unlocked`; and the compatibility section states the Rust-level breaks instead of calling the change additive. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015cD8PEeUfrzpqYuU1jaZBq
0047 and 0048 are allocated by PR #606, so the RFC becomes 0049 and the registry's next number 0050. From review: the ledger restore seam is withdrawn, since its real use is a coherent restore point where the ledger and the graphs come back together; readiness reports counts only and the graph ids stay behind the authenticated GET /graphs; the shutdown watchdog is an operating-system thread armed by a listener installed before graphs open; the authority enum gains `unlocked`; and the compatibility section states the Rust-level breaks instead of calling the change additive. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015cD8PEeUfrzpqYuU1jaZBq
rfc: accept RFC 0049 Maintainer decision recorded in the decision log; the registry row moves to accepted. Implementation status advances with the two implementation PRs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015cD8PEeUfrzpqYuU1jaZBq rfc: renumber to 0049, drop the ledger restore seam, record the review 0047 and 0048 are allocated by PR #606, so the RFC becomes 0049 and the registry's next number 0050. From review: the ledger restore seam is withdrawn, since its real use is a coherent restore point where the ledger and the graphs come back together; readiness reports counts only and the graph ids stay behind the authenticated GET /graphs; the shutdown watchdog is an operating-system thread armed by a listener installed before graphs open; the authority enum gains `unlocked`; and the compatibility section states the Rust-level breaks instead of calling the change additive. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015cD8PEeUfrzpqYuU1jaZBq rfc: add RFC 0048, control-plane seams Four small, independently shippable contracts an external control plane needs from the cluster crate and the server: observe-only reads (`plan --observe`, `cluster observe`), a ledger restore (`cluster state restore`), a readiness witness (`GET /readyz`), and a bounded shutdown (`--shutdown-grace-seconds`). Nothing changes a storage format, the ledger, the lock, or the recovery protocol; RFC 0034 and 0035 stay independent. The next number becomes 0049: 0047 is held by an out-of-tree draft. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015cD8PEeUfrzpqYuU1jaZBq
2 participants
Two public draft RFCs for the search-contracts program, per the RFC-first process (GOVERNANCE.md, size L):
RFC 0047 — Search plan truth: makes the existing search surface honest without any format or syntax addition — retrieval stated in the compiled plan (
QueryIR::retrieval), scan-rooted target validation (T26),fuzzy()retirement (T25), projectable ranking metrics, deterministic tie order, loud/observable failure modes (warnings, metric and retrieval metadata, embedding coverage). Prototyped end to end on the retained branch of closed PR #595 (2,860 workspace tests green, all review findings resolved); the prototype is evidence, not a landing request.RFC 0048 — Search contracts and retrieval algebra: the remainder of the program, blocked on 0047 — three composable search contracts; schema-owned
@analyzedanalyzer/scorer profiles whose fingerprints include substrate implementation identity;Vector(dim, distance=)and mandatory@embed(model=)embedding-space identity;match_terms(mode:), exactknn, approximateann(oversample:), N-arm weightedrrf(arm(...)); typed metric domains; deprecation ofsearch/match_text/nearest; and the accepted-SchemaIR/format rebuild boundary this requires. Backed by a Lance 11 change-by-change impact analysis (including an empirically confirmed stemmer-drift reproduction) and a fourteen-system constraint-placement survey.Both RFCs follow the template exactly and each ends with a non-normative agent context appendix carrying the implementation-facing dependency map, validated substrate assumptions, cross-RFC composition contracts (0040/0043/0044/0046), and CI traps — so human reviewers get the normative sections and coding agents get the deeper context in one place.
Registry updated: rows for 0047 and 0048, next available number 0049.
The RFCs request no code change; acceptance of each unlocks its staged implementation PRs as described in their Rollout sections.