docs: differentiator-forward README + refresh stale docs

[Fieldnote-Echo]

Summary

Refreshes the public docs for the sneak preview: a differentiator-forward README front door and a sweep of stale/incorrect numbers.

README (608 → 394 lines)

• Leads with the differentiators (tamper-evident single file, edge/tiny-RAM, no-build/no-training/no-BLAS, retrieval-quality parity, portable). Badges + LICENSE kept; working Rust/Python quickstarts preserved.
• Recall framing corrected. Dropped the exact-geometry claim ("returns the same top-k row IDs as an exact fp32 cosine scan — bit-identical"). Replaced with the true, stronger framing: OrdinalDB matches exact dense nDCG@10 on the public BEIR harness (lossless at bits=4, within bootstrap noise at bits=2). Exact ANN geometry (~0.82–0.87 recall by design) is not the retrieval-relevant metric.

Docs

• docs/limits.md — removed the obsolete 64 MB aux-sidecar cap and 524k-row cap (both lifted); core-vs-adapter scale clarified (core 1M+ single-file; ~100k is the adapter-directory planning envelope).
• docs/persistence.md — scale reference updated.
• docs/api.md — fixed a broken README.md#filter-dialects cross-reference to a section the rewrite removed.
• docs/matryoshka.md — dropped unsourced / self-contradicting recall figures; guidance now says measure-your-own.
• docs/roadmap/ordinaldb-trust-spec.md — removed an unbacked "Neither Chroma nor LanceDB offers this" line (it's a spec, not a marketing doc).

Guardrails

• No fiction: only in-repo / reproducible-harness numbers.
• No moat leaks: retrieval mechanism not disclosed; the banned exact-ANN-geometry-as-recall-metric removed.

Review focus

• README voice + the recall framing (this is the Paolo-facing front door)
• Confirm the nDCG-parity phrasing is scoped how you want it

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[qodo-code-review]

PR Summary by Qodo

Docs: differentiator-forward README and refresh stale documentation

📝 Documentation 🕐 20-40 Minutes

AI Description

• Rewrite README to lead with differentiators and tighter positioning
• Correct retrieval-quality framing (nDCG@10 parity) and remove unbacked recall claims
• Sweep docs to update scale limits, fix broken links, and remove unsourced assertions

Diagram

graph TD
R["README.md"] --> L["docs/limits.md"] --> P["docs/persistence.md"]
R --> A["docs/api.md"] --> M["docs/matryoshka.md"]
R --> T["docs/roadmap/ordinaldb-trust-spec.md"]
R --> B{{"BEIR harness"}}

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Split into two PRs (README rewrite vs. factual doc corrections)

• ➕ Easier reviewer focus: voice/positioning review separated from correctness review
• ➕ Reduces chance of missing a subtle factual change in a large rewrite
• ➖ Temporary inconsistency between README claims and supporting docs
• ➖ More process overhead and coordination between merges

2. Add a dedicated “Metrics & Methodology” doc and link from README

• ➕ Centralizes evaluation framing (nDCG vs geometry recall) and methodology details
• ➕ Reduces pressure to keep nuanced measurement text inside README
• ➖ Adds another top-level doc to maintain
• ➖ Still requires careful README summarization to avoid overclaiming

3. Keep adapter filter/operator matrix in README (appendix style)

• ➕ More self-contained for users skimming README only
• ➕ Reduces context switching while evaluating adapter support
• ➖ README grows and becomes harder to keep polished as the front door
• ➖ Higher likelihood of staleness vs. per-adapter/module documentation

Recommendation: The PR’s approach (README as a differentiator-forward entry point + moving details into docs + removing unbacked claims) is directionally correct and improves correctness. If review bandwidth is tight, consider the two-PR split as a process alternative; otherwise, merge as-is and optionally follow up with a single “Metrics & Methodology” doc to keep evaluation nuance out of the README while preserving reproducibility and scope control.

Files changed (6) +214 / -485

Documentation (6) +214 / -485

README.mdRewrite README front door and correct retrieval-quality framing +152/-360

Rewrite README front door and correct retrieval-quality framing

• Reorganizes the README to lead with differentiators (single-file verifiable bundle, edge deployment, no training/BLAS, portability) and clarifies scale guidance (core 1M+ single-file vs adapter-directory planning ~100k). Replaces the prior exact-geometry/bit-identical top-k claim with a nDCG@10 parity framing and adds tighter, scoped wording around bits/quality knobs, adapter behavior, and ops/persistence pointers.

README.md

api.mdFix stale README cross-reference and simplify hybrid section naming +15/-48

Fix stale README cross-reference and simplify hybrid section naming

• Removes a broken link to a README anchor that no longer exists and points readers to per-adapter module documentation instead. Renames/reframes the hybrid section to “Hybrid Search” and trims LTR feature-flag details to match the new top-level narrative.

docs/api.md

limits.mdUpdate limits narrative and remove obsolete auxiliary-size cap +24/-43

Update limits narrative and remove obsolete auxiliary-size cap

• Clarifies methodology/target wording and adds explicit separation between core single-file scale (1M+ rows) and adapter-directory planning guidance (~100k rows). Removes outdated constraints (e.g., a flat auxiliary-artifact cap) and updates guidance text accordingly.

docs/limits.md

matryoshka.mdRemove unsourced recall figures; shift to construction-based guidance +16/-30

Remove unsourced recall figures; shift to construction-based guidance

• Replaces small-corpus measured recall table and specific numbers with a construction-based explanation of why renormalization is a no-op for rank/sign coding. Reframes truncation tradeoffs as model/corpus-specific and recommends measuring recall for the user’s own setup.

docs/matryoshka.md

persistence.mdRefresh version references and clarify core vs adapter scale +6/-3

Refresh version references and clarify core vs adapter scale

• Updates artifact/version references (e.g., ordvec-manifest version) and tightens language around current artifact filenames. Expands the adapter save-cost section to explicitly scope the ~100k planning guidance to adapter directories and distinguish it from the core single-file '.odb' path scaling to 1M+ rows.

docs/persistence.md

ordinaldb-trust-spec.mdRemove unbacked competitor comparison from trust spec +1/-1

Remove unbacked competitor comparison from trust spec

• Drops a specific claim about competitor capabilities to keep the trust spec focused on design intent and defensible statements. Leaves the rest of the design sketch intact.

docs/roadmap/ordinaldb-trust-spec.md

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Single-file claim incorrect ✓ Resolved 🐞 Bug ≡ Correctness

Description

README.md now advertises OrdinalDB as a “single verifiable file,” but the .odb format is a
directory bundle (manifest + artifacts) per the persistence layout and OrdinalIndex::write/load
docs. This is a factual mismatch in the project’s front-door differentiator and can mislead users
about portability/backup expectations.

Code

README.md[R6-12]

+**Vector search you can ship as a single verifiable file.**
+
+OrdinalDB is a training-free, tamper-evident, embedded vector database. Bring
+your own embeddings, index the very first vector with zero prior data, and
+write the whole index as one self-contained, SHA-256-manifested `.odb` bundle
+that scales past a million rows and opens verified anywhere. No server, no
+BLAS, no learned codebook, no ANN graph — and no recall knob to get wrong.

Relevance

⭐⭐⭐ High

Team often accepts doc accuracy fixes; .odb treated as directory in examples. Similar doc-mismatch
fixes accepted.

PR-#36
PR-#31
PR-#30

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The README introduces the “single verifiable file” claim, but both the README persistence layout and
the Rust API docs describe .odb as a directory bundle, proving the wording is
incorrect/inconsistent with the implementation.

README.md[6-12]
README.md[312-320]
ordinaldb/src/ordinal.rs[1042-1102]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The README claims the index is a **single verifiable file**, but the actual `.odb` persisted format is a **bundle directory** (contains `manifest.json`, `index.ovrq`, `sign.ovsb`, etc.). This creates a user-visible factual inconsistency.

### Issue Context
- The README headline/intro now emphasizes “single file”.
- The same README’s persistence section shows `docs.odb/` as a directory.
- The Rust API docs explicitly describe writing/loading a **bundle directory**.

### Fix Focus Areas
- README.md[6-12]
- README.md[310-320]
- ordinaldb/src/ordinal.rs[1042-1102]
- docs/limits.md[18-26]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Unsubstantiated recall claim ✓ Resolved 🐞 Bug ≡ Correctness

Description

docs/limits.md asserts top‑k row IDs are bit-identical to an exact fp32 cosine brute-force baseline,
but the referenced benchmark method (scripts/limits_report.py) does not compute or compare against
any brute-force cosine baseline. This makes the recall statement unsupported by the cited
measurement.

Code

docs/limits.md[R28-30]

+- **Recall:** top-k row ids are bit-identical to an exact fp32 cosine
+  brute-force baseline on the measured query sets. There is no
+  recall/ef/nprobe knob to get wrong.

Relevance

⭐⭐⭐ High

Historical precedent: they accept narrowing docs when claims exceed measured methods; similar “doc
claim vs code” fix accepted.

PR-#30

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The limits doc states the method is scripts/limits_report.py and then makes a brute-force cosine
equivalence claim. The script only builds/writes/loads/searches and times operations; it never
computes a brute-force cosine ranking nor compares returned IDs, so the claim is not established by
the cited method.

docs/limits.md[3-5]
docs/limits.md[28-30]
scripts/limits_report.py[125-153]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`docs/limits.md` claims exact agreement with an fp32 cosine brute-force baseline, but the documented method (`scripts/limits_report.py`) does not perform any baseline computation or ID comparison. The claim should be removed/softened, or the benchmark script and report generation should be extended to actually measure it.

### Issue Context
The doc explicitly ties its claims to `scripts/limits_report.py` as the measurement method.

### Fix Focus Areas
- docs/limits.md[3-5]
- docs/limits.md[18-34]
- scripts/limits_report.py[125-153]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---