Wave 1e: Query engine (n-gram extraction, posting list intersection)

[leesharon]

Overview

Query engine: text query → n-grams → index lookup → posting list intersection → BM25F scoring → ranked results.

Module: crates/rskim-search/src/lexical/query.rs (~200 lines)

Pipeline

Query string → extract_query_ngrams → lookup each in index (binary search .skidx) → load posting lists (mmap .skpost) → intersect (sorted merge) → score (BM25F) → sort by score → top-K

Optimizations

• Process rarest n-gram first (smallest posting list) — "cheapest first" strategy
• Lazy posting list iteration (never fully materialize)
• Early termination when top-K scores can't be exceeded

Dependencies

• Wave 1a: Sparse n-gram algorithm (bigram weights, covering set) #5 (n-gram extraction), Wave 1b: Two-file mmap'd index format (lookup table + postings) #6 (index format), Wave 2a: Git log parser via gix (commits, changed files, messages) #7 (BM25F scoring)

Acceptance Criteria

• Query "handleRequest" returns files with that identifier in top 3
• Query latency < 50ms for 10k-file index
• Empty query returns empty results (not error)
• Unicode queries work
• Results include file path, line, score, snippet

Implementation Guidelines Checklist

• No file exceeds 400 lines
• Every public function has /// doc comment
• All dependencies injected
• All fallible operations return Result<T, SearchError>
• Module has #[cfg(test)] mod tests with: happy path, edge case, error path
• No hardcoded language-specific match arms
• No Vec::clone() in hot paths
• Posting list intersection uses lazy evaluation