[core] Rerank primary-key vector candidates with exact distances

[JunRuiLee]

Purpose

Part of #514. Adds exact rerank to primary-key vector search, mirroring Java PrimaryKeyVectorRead's refine/rerank. On a PK-vector table, execute_read() materializes rows in file-local physical coordinates; this lets a configured refine factor over-fetch approximate (ANN) candidates and reorder them by exact distance for higher recall, while leaving the no-refine path byte-identical.

Brief change log

• Dual-limit bucket search: bucket_search returns separate indexed/exact candidate lists, each bounded independently — the ANN/indexed channel over-fetches (indexed_limit = limit × refine_factor) while the exact-fallback channel stays bounded by the final limit. PkVectorOrchestrator::search_candidates takes the dual limits and returns an OrchestratorSearchResult; merge_candidates runs a global Top-K over the union of the two channels.
• Rerank: when a positive refine factor is configured, rerank_indexed_positional rereads the over-fetched ANN candidates' vectors from their data files by file-local position (reusing the position read path), recomputes exact distance, and re-runs the global Top-K. Exact-fallback candidates are already exact and are not reranked.
• Refine-factor resolution reuses the existing option-key precedence (refine_factor / refine-factor / rerank_factor / rerank-factor, with fields.<col>. → index-type → ivf. → bare prefix order); factor 0 (unset) leaves indexed_limit == limit, so output is byte-identical to the no-rerank path.
• Fail-loud validation: append_batch_vectors rejects null child elements in vector arrays (never a defaulted 0.0), build_indexed_splits validates split_index is in range before indexing, and the rerank kernel fails loud on ownership/dedup/NULL/dimension/file-not-in-plan violations — never a panic or a silently corrupted result.

Tests

• Dual-limit bucket behavior; merge_candidates union Top-K; rerank kernel (ownership order, dedup, NULL element, dimension mismatch, file-not-in-plan); end-to-end refine integration matching an exact brute-force ground truth.
• Regression invariant: refine_factor == 0 preserves the exact pre-rerank output (baseline + Java-fixture read-back tests unchanged).
• Fail-loud coverage: null vector element, out-of-range split_index.
• cargo test -p paimon green; cargo build -p paimon-datafusion clean (cross-crate Send gate); cargo clippy -D warnings and cargo fmt --check clean.

API and Format

No on-disk format change, no new result columns. refine_factor == 0 (the default) is byte-identical to the current output.

Documentation

Code comments only; no user-facing docs change.

[leaves12138]

The rerank implementation and focused tests look solid overall, but refine-factor validation is currently data-dependent: an empty PK-vector plan returns before parsing the option. Please address the inline finding so invalid query/table configuration fails consistently.

[leaves12138]

configured_refine_factor is reached only after PkVectorScan::plan() and the plan.splits.is_empty() early return above. As a result, an empty PK-vector table accepts refine_factor=0 or refine_factor=abc and returns a normal empty reader, while the identical query starts failing as soon as the table has a searchable split. Java resolves this option in the PrimaryKeyVectorRead constructor before planning, and configuration validity should not depend on whether data currently exists. Please resolve the factor/search limit before the empty-plan return (ideally before planning/index I/O) and add an empty-plan regression test.

[JunRuiLee]

Fixed in 1ab7337. configured_refine_factor / indexed_search_limit are now resolved before PkVectorScan::plan() and the empty-plan early return, so an invalid factor fails loud regardless of whether the table currently has searchable data — matching Java, which resolves this in the PrimaryKeyVectorRead constructor before planning. Added an empty-table regression test covering both invalid forms you named (0 → "must be positive", and a non-integer → "must be an integer") and both option sources (query and table options, since the fix moved resolution of both ahead of planning).

[leaves12138]

Re-reviewed at 51c5c42. The refine factor and derived indexed search limit are now resolved before planning and the empty-plan return, so invalid query or table configuration fails consistently; the new empty-table regression covers the reported cases. The dual-limit search and exact positional rerank remain aligned with the Java flow. Verified locally with git diff --check, cargo fmt --check, cargo test -p paimon --lib (1705 passed, 1 ignored), cargo test -p paimon --test pk_vector_baseline_test (8 passed), and cargo check -p paimon-datafusion.