fix(query): push the limit clause down into bm25 full-text scans

[azimafroozeh]

What & why

Closes #563. A bm25() or rrf() ordered read that also traverses an edge failed with an Arrow Offset overflow error at corpus scale (reported at ~700k+ entities), because the two ranked orderings were asymmetric: nearest() threads the query limit into the scan as top-k, while the bm25 leg never set the full-text query's limit, so Lance returned every matching entity ranked and the pipeline materialized the whole matched set, applying limit only at the very end. With a traversal, the joined text column was re-copied at edge fanout and crossed the 2 GiB i32 offset ceiling of a single Arrow string column; without one, a limit-20 read still hydrated the full matched corpus (the reporter measured 71 s at 1.05M entities).

Measured with the timing instrument this PR adds to the repro target (6,000 matched entities x 200 KB text, warm, identical debug builds of the parent commit and this branch): the join-free limit-20 read drops from 119 ms to 2 ms median. The cost is hydration count (6,000 entities before, 80 after), so the gap widens with matched-corpus size.

The fix makes the bm25 leg carry the limit the way nearest always has, plus a safety net so the bound can never change an answer:

• The full-text scan is capped at four times the limit: BM25 returns entities score-descending, so the capped scan yields the uncapped scan's leading entities (up to score ties), and the slack absorbs entities later dropped by traversals or non-pushed filters.
• If the capped pass still returns fewer than limit results, the query retries once uncapped: the cap is an optimization, never a result budget. Retries emit a debug trace event.
• Aggregate returns are never capped: an aggregate's value is computed over the scanned entities, so a cap would change count/sum answers, not just cost (pinned by a test that failed red before this exemption, and by probe assertions proving the aggregate's single scan is uncapped).
• Both rrf() arms are now bounded. Previously the vector arm was capped at the limit while the FTS arm was exhaustive, so fusion was already one-sided top-N; this makes the arms consistent, at the cost that entities ranked past an arm's cap lose that arm's fusion contribution (tail ordering on large matched sets can differ from before; documented in the search docs).
• Because capped and uncapped runs are otherwise result-identical, the tests assert the mechanism through two test-only probes: a retry counter and a scanned-rows counter that pins the cap's magnitude (a silent factor regression turns a pinned count red in both directions).

Backing issue / RFC

• none; bug fix motivated by performance: bm25/rrf ranked read + a join materializes the joined column corpus-wide → Arrow 2 GB offset overflow at ~700k+ rows #563 (the column-width half of the same report is performance: projecting non-vector columns still reads the full Vector column — read cost/OOM scale with the embedding column even when unused #564)

Checklist

• Change is focused (bound the ranked FTS scan and add the under-fill retry; nothing else)
• Tests added/updated for behavior changes (three always-on mechanism tests with retry-fired and scan-row-magnitude probe assertions, incl. a red-first aggregate exemption test; an #[ignore]d overflow-scale repro plus a timing instrument in the same target)
• Public docs updated if user-facing surface changed (docs/user/search/index.md gains the bounded-scan paragraph incl. the rrf fusion-window note)
• Reviewed against docs/dev/invariants.md — no Hard Invariant weakened, no deny-list item hit (the general read-path memory bound remains absent; this closes the ranked-read instance only)

Local verification

• cargo test -p omnigraph-engine --test search — 34 passed
• cargo test -p omnigraph-engine --test repro_issue_563 -- --ignored --nocapture — both pass: the overflow repro returns its 20 rows in ~21 s (pre-fix: Offset overflow error: 2147489268), the timing instrument prints 2 ms median (119 ms on the parent commit, same instrument via a git-archive build)
• cargo test --workspace --no-fail-fast — green except three failures reproduced identically on pristine main (sandboxed special-file blob test, two merge stack overflows)
• cargo clippy --workspace --all-targets — clean
• cargo fmt --all --check — clean

Notes for reviewers

• A query whose complete answer is legitimately smaller than its limit pays the capped-plus-uncapped double run on every execution: the result count cannot distinguish cap starvation from a small matched set. A scan-fill signal to skip the futile retry is deliberate future work.
• The cap bounds scan results only: traversal fanout and search() filters without a bm25 ordering still materialize unbounded (the pre-existing read-path memory-bound class), and at overflow scale a genuinely under-filled query still errors in the uncapped retry, as before this change.
• The overfetch factor is a named constant, not configuration: exposing it waits for the retry telemetry this PR adds to show real tuning demand.
• Pre-existing and unchanged: a declared asc/desc on a bm25() ordering is ignored engine-wide (the cap assumes the score-descending order Lance actually returns), and the rrf k positivity check accepts only literal integers, so a param-supplied non-positive k reaches runtime; this PR makes that cast saturate deterministically instead of wrap.

Greptile Summary

This PR bounds standalone BM25 scans according to the query limit while preserving complete results through an uncapped retry when downstream filtering or traversal under-fills the result.

• Exempts aggregate projections and RRF full-text arms from scan caps.
• Adds query instrumentation and tests covering capped scans, retries, aggregates, RRF ranking, and the corpus-scale overflow reproduction.
• Documents the behavior in the user guide and unreleased v0.11.0 release notes.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
crates/omnigraph/src/exec/query.rs	Introduces the BM25 scan cap, aggregate exemption, under-fill retry, and explicit uncapped RRF full-text arms.
crates/omnigraph/src/exec/projection.rs	Extracts aggregate-projection detection for reuse when deciding whether a BM25 scan may be capped.
crates/omnigraph/src/instrumentation.rs	Adds test probes for uncapped retries and BM25 scan-row counts.
crates/omnigraph/tests/search.rs	Adds mechanism tests for capped scans, retry behavior, aggregate correctness, and complete RRF rankings.
crates/omnigraph/tests/repro_issue_563.rs	Adds ignored corpus-scale correctness and timing reproductions for the Arrow offset-overflow scenario.
docs/releases/v0.11.0.md	Records the user-visible ranked-scan behavior in the current unreleased release notes.
docs/user/search/index.md	Documents bounded standalone BM25 scans and why RRF full-text arms remain uncapped.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    Q[BM25 ordered query] --> A{Aggregate return?}
    A -->|Yes| U[Run uncapped BM25 scan]
    A -->|No| L{Limit present?}
    L -->|No| U
    L -->|Yes| C[Run capped BM25 scan]
    C --> P[Traverse filter project order and limit]
    P --> F{Returned fewer rows than limit?}
    F -->|No| R[Return result]
    F -->|Yes| U2[Retry once with uncapped scan]
    U2 --> R
    U --> P2[Execute remaining pipeline]
    P2 --> R

Loading

_{Reviews (5): Last reviewed commit: "Merge remote-tracking branch 'upstream/m..." | Re-trigger Greptile}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[aaltshuler]

Verified one ranking regression against the exact PR head and its parent; details inline.