docs: changelog perf entries + 0.6.0 un-patch checklist

[Fieldnote-Echo]

CHANGELOG ### Performance entries for the four unlogged perf changes (numbers verified against source commit messages) and the RELEASING.md one-time un-patch step for ordinaldb's [patch.crates-io] block. Fleet-authored, reviewed APPROVE. Merge LAST — cut from the integration branch, so its commit list collapses to docs-only once #277/#278/#279/#280/#281 land. Known follow-up: a CHANGELOG line for the CLI-parity PR once it merges.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[qodo-code-review]

PR Summary by Qodo

Add performance changelog entries and implement derived artifact size bounds

✨ Enhancement 🐞 Bug fix 🧪 Tests 📝 Documentation 🕐 40+ Minutes

AI Description

• Document previously-unlogged performance work and add a one-time 0.6.0 release hygiene step.
• Make ordvec-manifest artifact read limits derived from declared sizes, with opt-in ceilings.
• Speed up hot paths: streamed batched candidate generation, parallel finite checks, LUT validation.
• Add contract-pinning and regression tests for streaming candgen and derived-size enforcement.

Diagram

graph TD
A["Public APIs"] --> B["SignBitmap candgen"] --> C["Streamed top-m core"]
A --> D["RankQuant"] --> E["Load/encode perf"]
A --> F["ordvec-manifest"] --> G["Bounded hashing/limits"]
H["Docs"] --> A

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Keep flat default caps; only raise the 64 MiB defaults

• ➕ Minimal behavioral change for deployments relying on implicit ceilings
• ➕ Less surface area for new limit interactions (declared-size vs cap)
• ➖ Still couples correctness to an arbitrary default; legitimate artifacts can regress again
• ➖ Doesn't address unbounded primary-artifact hashing unless separately handled

2. Stream candgen but use per-query partial sort on chunked score buffers

• ➕ Could retain O(n) average selection per query via select_nth-like approaches
• ➕ May reduce heap overhead for large m
• ➖ More complex to implement correctly with global (hamming, doc_id) ordering across blocks
• ➖ Harder to make processing-order-independent without careful merging logic

Recommendation: The PR’s approach is the better long-term contract fit: (1) derived-size read bounds make manifests the trust anchor while still allowing opt-in ceilings for hostile-input deployments, and (2) streamed candidate generation removes pathological re-scans while preserving a strict, test-pinned ordering contract. The included oracle/edge-case suites materially de-risk the algorithmic changes, making this strategy preferable to simply bumping flat defaults or introducing more complex block-merge selection logic.

Files changed (13) +824 / -49

Enhancement (4) +156 / -17

quant.rsReuse per-worker rank scratch during RankQuant::add +10/-7

Reuse per-worker rank scratch during RankQuant::add

• Switches the parallel encode loop to 'for_each_init' and reuses a per-worker 'Vec<u16>' ranks buffer. Uses 'rank_transform_into' to avoid per-vector allocation churn inside the hot loop.

src/quant.rs

rank_io.rsSpeed up RankQuant load validation with LUT + parallel scan +34/-2

Speed up RankQuant load validation with LUT + parallel scan

• Adds a per-byte bucket-count LUT to validate constant-composition rows using table lookups instead of per-code shift/mask loops. Parallelizes the row validation with 'find_first' to preserve the 'lowest offending row' contract and re-runs the scalar histogram only for error reporting.

src/rank_io.rs

sign_bitmap.rsImplement streamed batched top-m candidate generation +101/-7

Implement streamed batched top-m candidate generation

• Introduces a new serial streamed core that scans the corpus once per call in doc blocks, scores query tiles, and keeps exact per-query top-m using bounded heaps keyed by (hamming, doc_id). Optimizes collector acceptance via cached worst-hamming bounds and updates batched CSR generation to route through the streamed core (with explicit empty-batch handling) while keeping single-query on the dense partition path for performance.

src/sign_bitmap.rs

util.rsParallelize assert_all_finite for large inputs +11/-1

Parallelize assert_all_finite for large inputs

• Adds a size threshold to scan large float slices in parallel using rayon chunks, reducing validation overhead for large ingest/search batches. Keeps the existing fast serial path for smaller inputs.

src/util.rs

Bug fix (2) +93 / -19

lib.rsDerive artifact read bounds and stream bounded hashing +72/-12

Derive artifact read bounds and stream bounded hashing

• Changes default ResourceLimits byte caps to 'u64::MAX' (unbounded) and introduces 'max_index_artifact_bytes' as an opt-in ceiling. Updates verification and SQLite-related hashing call sites to bound reads by 'min(declared_size, configured_ceiling)' and to fail fast with '*_file_too_large' codes. Reimplements 'sha256_file_bounded' as a streaming reader to keep memory constant and bounds create-time hashing by observed file length.

ordvec-manifest/src/lib.rs

sqlite.rsBound SQLite cache-key hashing like verify path +21/-7

Bound SQLite cache-key hashing like verify path

• Removes unbounded 'sha256_file' usage for artifact cache keys and switches to 'sha256_file_bounded' using declared size with the opt-in ceiling. Updates profile hashing helpers to use declared-size-derived bounds as well, ensuring cache behavior matches verification semantics.

ordvec-manifest/src/sqlite.rs

Tests (3) +473 / -9

derived_limits.rsAdd end-to-end tests for derived-size limits and opt-in ceilings +223/-0

Add end-to-end tests for derived-size limits and opt-in ceilings

• Introduces a new test module asserting that default options accept artifacts above the legacy 64 MiB cap, while verify fails fast when artifacts grow beyond declared size. Adds regression coverage for truncated/inflated declarations and confirms explicit flat caps (including the new index ceiling) remain enforceable.

ordvec-manifest/tests/derived_limits.rs

manifest.rsAdjust artifact corruption tests and add grown-profile fast-fail coverage +75/-9

Adjust artifact corruption tests and add grown-profile fast-fail coverage

• Updates a corruption test to flip bytes in-place (same size) so the declared-size bound passes and the digest mismatch is exercised. Updates the 'not a byte pin' test to expect 'artifact_file_too_large' when the artifact grows past its declaration. Adds a new regression test ensuring grown calibration/distortion profile files fail fast with the *_too_large codes under default limits.

ordvec-manifest/tests/manifest.rs

tiled_candgen.rsAdd oracle-based contract tests for candidate generation ordering +175/-0

Add oracle-based contract tests for candidate generation ordering

• Adds an independent oracle (score_all + full lexicographic sort by (hamming asc, doc_id asc)) to pin 'top_m_candidates' and 'top_m_candidates_batched_serial_csr' outputs exactly. Covers random corpora across block boundaries, tie-heavy and duplicate-run cases, empty batches, and high-dimension shapes (1024 and 768 tail residue).

tests/tiled_candgen.rs

Documentation (4) +102 / -4

CHANGELOG.mdAdd missing Unreleased performance and manifest-limit entries +75/-1

Add missing Unreleased performance and manifest-limit entries

• Replaces the empty Unreleased section with detailed Performance bullets describing streamed batched candidate generation, collector accept-test optimization, parallel finite validation + RankQuant scratch reuse, and LUT-based RankQuant load validation. Adds Changed entries documenting derived artifact size bounds, new opt-in index artifact ceiling, fail-fast reason-code behavior, and constant-memory bounded hashing.

CHANGELOG.md

RELEASING.mdAdd one-time 0.6.0 downstream un-patch checklist step +5/-0

Add one-time 0.6.0 downstream un-patch checklist step

• Documents that OrdinalDB must remove its '[patch.crates-io]' overrides for 'ordvec' and 'ordvec-manifest' upon the 0.6.0 publish. Clarifies this is a one-time release hygiene action so downstream consumes crates.io releases rather than a git branch.

RELEASING.md

THREAT_MODEL.mdDocument derived artifact read bounds threat +13/-0

Document derived artifact read bounds threat

• Adds THREAT-QUERY-003 describing that verification work scales with manifest-declared bytes even with constant-memory hashing. Explains the role of opt-in ResourceLimits ceilings and reiterates that VerifiedLoadPlan is not a byte pin under local write access.

THREAT_MODEL.md

README.mdUpdate limit contract docs for derived-size bounds +9/-3

Update limit contract docs for derived-size bounds

• Rewrites artifact byte-limit bullets to state that reads are bounded by manifest-declared 'file_size_bytes' on verify and observed size on create. Adds the primary index artifact bound and clarifies flat caps are opt-in ceilings with unbounded defaults.

ordvec-manifest/README.md

[codecov]

Codecov Report

❌ Patch coverage is 97.88732% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/sign_bitmap.rs	98.09%	2 Missing ⚠️
src/rank_io.rs	95.83%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Bounded hash reads overshoot ✓ Resolved 🐞 Bug ≡ Correctness

Description

sha256_file_bounded enforces max_bytes only after reading a full 64KiB chunk, so it may read up
to ~64KiB past the declared/configured bound (and reads 64KiB even when max_bytes=0). This breaks
the intended strict read-bound contract and causes unnecessary extra I/O under tight limits.

Code

ordvec-manifest/src/lib.rs[R3486-3506]

+    let mut file = File::open(path)?;
    let mut hasher = Sha256::new();
-    hasher.update(&bytes);
+    let mut size_bytes = 0u64;
+    let mut buf = [0u8; 64 * 1024];
+    loop {
+        let n = file.read(&mut buf)?;
+        if n == 0 {
+            break;
+        }
+        size_bytes += n as u64;
+        if size_bytes > max_bytes {
+            return Err(ManifestError::limit_exceeded(
+                code,
+                format!(
+                    "{context} exceeds {max_bytes} bytes while reading {}",
+                    path.display()
+                ),
+            ));
+        }
+        hasher.update(&buf[..n]);
+    }

Relevance

⭐⭐⭐ High

Team previously enforced strict bounded reads via take(max_bytes+1) in read_bounded_file; likely
expects same for hashing.

PR-#157

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

sha256_file_bounded reads fixed 64KiB chunks and only checks size_bytes > max_bytes after
incrementing, so the function can consume bytes past the limit before failing. The repo’s
read_bounded_file demonstrates the strict-bounding approach using take(max_bytes + 1),
highlighting the inconsistency with the new hashing implementation.

ordvec-manifest/src/lib.rs[3479-3511]
ordvec-manifest/src/lib.rs[138-176]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`sha256_file_bounded` can read beyond `max_bytes` by up to one buffer (64KiB) because it reads into a fixed buffer and only checks `size_bytes > max_bytes` after the read. This violates a strict "bounded read" expectation (especially noticeable when `max_bytes` is very small).

### Issue Context
The codebase already uses a strict-bounding pattern for `read_bounded_file` via `take(max_bytes + 1)`.

### Fix Focus Areas
- ordvec-manifest/src/lib.rs[3479-3511]

### Implementation notes
Adjust the loop so the reader never consumes more than `max_bytes + 1` total bytes, e.g.:
- wrap the file with `file.by_ref().take(max_bytes.saturating_add(1))` and stream from that, or
- compute `remaining = max_bytes.saturating_sub(size_bytes)` and only attempt to read `min(buf.len(), remaining as usize + 1)` each iteration.
Ensure the error is still returned with the same `LimitExceeded` code/message when the bound is exceeded.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. TOCTOU size/hash mismatch ✓ Resolved 🐞 Bug ≡ Correctness

Description

create_manifest_for_index_with_options probes index metadata (including file_size_bytes) and
then hashes the index using a separately-fetched fs::metadata(index_path)?.len() bound; if the
file size changes between those calls, the manifest can embed a file_size_bytes that does not
match the bytes that were actually hashed. This can produce a manifest that fails verification
immediately or behaves inconsistently under concurrent writers.

Code

ordvec-manifest/src/lib.rs[R3566-3569]

+    let index_hash = sha256_file_bounded(
+        index_path,
+        fs::metadata(index_path)?.len(),
+        "artifact_file_too_large",

Relevance

⭐⭐⭐ High

Similar TOCTOU-hardening was definitely accepted: reuse verification-time resolved paths/metadata to
avoid same-call races.

PR-#163

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The create path uses probe_index_metadata() (whose file_size_bytes is derived from
file.metadata().len()) but hashes with a fresh fs::metadata(...).len() bound and then writes
metadata.file_size_bytes into the manifest, enabling divergence if the file size changes between
the two observations.

ordvec-manifest/src/lib.rs[3565-3586]
src/rank_io.rs[368-384]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`create_manifest_for_index_with_options` observes the index file size twice: once indirectly via `probe_index_metadata()` (stored into `metadata.file_size_bytes` and later written into the manifest), and again via `fs::metadata(index_path)?.len()` to set the hash read bound. If the file changes between these observations, the created manifest can contain a `file_size_bytes` that doesn't correspond to the hashed content.

### Issue Context
This is a classic TOCTOU inconsistency during manifest creation. The create path should either:
- use one size snapshot consistently for both hashing and the manifest field, or
- explicitly fail if the size changes between probe and hash.

### Fix Focus Areas
- ordvec-manifest/src/lib.rs[3565-3586]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Per-query bitmap allocations ✓ Resolved 🐞 Bug ➹ Performance

Description

SignBitmap::top_m_candidates_streamed builds q_bitmaps by calling build_query_bitmap per
query, which allocates a fresh Vec<u64> each time and re-validates finiteness, even though the
batch entry point already validates the whole query buffer. This adds avoidable allocation/CPU
overhead in the new hot-path implementation.

Code

src/sign_bitmap.rs[R241-245]

+        let mut q_bitmaps = vec![0u64; nq * qpv];
+        for qi in 0..nq {
+            let qb = self.build_query_bitmap(&queries[qi * dim..(qi + 1) * dim]);
+            q_bitmaps[qi * qpv..(qi + 1) * qpv].copy_from_slice(&qb);
+        }

Relevance

⭐⭐⭐ High

Repo has repeatedly accepted perf fixes removing avoidable allocations/copies in hot paths; similar
guidance accepted for SignBitmap/scan code.

PR-#96
PR-#208

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The streamed path allocates per query by calling build_query_bitmap inside a loop and then copies
into q_bitmaps. build_query_bitmap itself allocates a new Vec<u64> and runs
assert_all_finite, while top_m_candidates_batched_serial_csr already validates the full
queries slice before calling the streamed implementation.

src/sign_bitmap.rs[202-213]
src/sign_bitmap.rs[235-245]
src/sign_bitmap.rs[424-447]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The streamed batched candidate generator constructs query bitmaps via `build_query_bitmap` for each query. `build_query_bitmap` allocates a `Vec<u64>` and also calls `assert_all_finite`, which means the streamed path incurs per-query allocations and additional passes over query memory.

### Issue Context
`top_m_candidates_batched_serial_csr` already asserts `assert_all_finite(queries)` and has a preallocated `q_bitmaps` buffer sized `nq * qpv`.

### Fix Focus Areas
- src/sign_bitmap.rs[202-245]
- src/sign_bitmap.rs[424-447]

### Implementation notes
Introduce a private helper that fills an existing `&mut [u64]` (or `&mut Vec<u64>`) for a single query without allocating, e.g. `build_query_bitmap_into(q: &[f32], out: &mut [u64])`.
- In `top_m_candidates_streamed`, fill `q_bitmaps[qi*qpv..(qi+1)*qpv]` directly.
- Keep `build_query_bitmap` as the public allocating API, but avoid calling it from the streamed inner loop.
- Since the batch entry point already validated `queries`, avoid re-running `assert_all_finite` per query in the streamed path.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Artifact zero-size unvalidated ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

Verification now bounds primary-artifact hashing by manifest.artifact.file_size_bytes, but
validate_manifest_shape does not reject artifact.file_size_bytes == 0 (unlike
calibration/distortion profile artifacts). For malformed manifests that declare file_size_bytes=0
for a non-empty index, this shifts the failure into artifact_file_too_large instead of a clearer
schema/shape error, making reports less consistent and harder to interpret.

Code

ordvec-manifest/src/lib.rs[R264-273]

+        match sha256_file_bounded(
+            &resolved.canonical_path,
+            document
+                .manifest
+                .artifact
+                .file_size_bytes
+                .min(options.limits.max_index_artifact_bytes),
+            "artifact_file_too_large",
+            "index artifact",
+        ) {

Relevance

⭐⭐ Medium

Team accepts tightening verifier contracts, but no prior review explicitly required rejecting
artifact file_size_bytes==0.

PR-#157
PR-#152

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The verification path now uses artifact.file_size_bytes as the hash bound, but the manifest shape
validator checks many artifact fields and does not validate artifact.file_size_bytes. In contrast,
both profile artifact validators explicitly error when file_size_bytes == 0, showing an
inconsistency in schema enforcement.

ordvec-manifest/src/lib.rs[258-301]
ordvec-manifest/src/lib.rs[328-387]
ordvec-manifest/src/lib.rs[1200-1217]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
Primary artifact `file_size_bytes` is now used as the effective hash read bound, but the manifest shape validator does not assert it is > 0. This creates an inconsistency with profile artifacts (which explicitly reject zero) and can lead to confusing error codes for malformed manifests.

### Issue Context
A zero-sized primary index artifact is not a meaningful/valid index file in practice (it has at least a header), so treating `file_size_bytes == 0` as a schema/shape error keeps reporting consistent and clearer.

### Fix Focus Areas
- ordvec-manifest/src/lib.rs[328-387]
- ordvec-manifest/src/lib.rs[258-301]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---