perf(vector/search): naN handling in float heaps via `unwrap_or(Ordering::Equal)` — silent reorder risk

[mosuka]

Round-3 perf push sub-issue (tracked under umbrella #536).

[S] NaN handling in float heaps via unwrap_or(Ordering::Equal) — silent reorder risk

• Where: hnsw/searcher.rs:270-310, vector/store.rs:480-495, engine.rs:1485. All
  use partial_cmp(...).unwrap_or(Ordering::Equal).
• Current behavior: A NaN distance gets treated as equal to everything → heap invariant
  silently broken → possible wrong top-K. Kernel emits NaN-free values today but a single
  bad input via fallback path would corrupt the heap.
• Reference precedent: ordered_float::NotNan; Lucene asserts non-NaN; Qdrant uses
  f32::total_cmp.
• Suggested direction: Use f32::total_cmp (Rust 1.62+) for deterministic order, or
  validate at kernel boundary. Wrap heap items in NotNanF32 newtype. With the packed
  candidate (above), this is automatic.
• Risk / scope: Small.

---

ID: VS-24 — see ~/.claude/tasks/laurus/20260523_perf_round3_audit/task_list.md for the full Round-3 issue list.

[mosuka]

Investigation & plan (phase sync)

Investigation

NaN-unsafe float ordering is widespread: 31 partial_cmp(...).unwrap_or(Ordering::Equal) sites across 17 files, plus 9 partial_cmp(...).unwrap() sites that panic on NaN. sort_unstable_by / BinaryHeap require a total order; unwrap_or(Equal) treats NaN as equal to everything (breaking the total-order contract → silent reorder, or a panic on newer std), and the .unwrap() variants panic on NaN outright. NaN scores arise from degenerate vectors and 0/0 (e.g. deletion_ratio when total_docs == 0).

The "float heaps" core is HNSW Candidate/ResultCandidate (hnsw/searcher.rs:419/449), whose Ord::cmp uses partial_cmp(...).unwrap_or. bkd_tree.rs:454-459 already uses total_cmp as precedent.

Plan

Replace all NaN-unsafe float comparators with f32::total_cmp / f64::total_cmp (a real IEEE-754 total order):

1. Sort comparators → total_cmp, preserving the existing a/b order (direction unchanged).
2. Heap / sort-key Ord::cmp impls (HNSW Candidate/ResultCandidate, collector sort-key structs) → total_cmp on the float field, keeping min/max-heap semantics.
3. FieldValue::Float64 comparison in the sort-by-field path → total_cmp.
4. .unwrap() panic sites: score/distance (advanced_query.rs:213, geo.rs:580) and segment-maintenance priority / deletion_ratio (merge_policy/optimization/manager/merge_engine) → total_cmp.

Non-float partial_cmp (Text/Int64/Bool/DateTime .cmp) is out of scope (already total). For non-NaN values total_cmp matches partial_cmp exactly, so behaviour is unchanged except NaN is now ordered deterministically instead of reordering/panicking. Public API unchanged.

Out of scope (follow-up): type-level enforcement via an OrderedFloat wrapper for float BinaryHeaps.

Local docs (Japanese) under ~/.claude/tasks/laurus/20260603_nan_safe_total_cmp/.