[core] Optimize vector refine raw reads

[JingsongLi]

Summary

Optimize vector raw refine by keeping candidate bitmaps as the internal source of truth, reading only the columns needed for refine, and maintaining bounded topK state while scoring. Batch vector refine now unions candidates across queries and reads raw vectors once before scoring each query's own candidate set.

Changes

• Keep refine candidates as bitmap-first inputs and use row ranges only as the current read-plan selection mechanism.
• Score only rows present in the candidate bitmap, which prepares the path for future wider raw read windows without changing scoring semantics.
• Maintain topK heaps during raw scoring instead of materializing all candidate scores and trimming afterward.
• For batch vector refine, read the union of candidate raw vectors once and rerank per query against its own candidates.
• Align Python vector search refine behavior with the Java implementation.
• Add tests for refine projection, candidate-only scoring, batch union reads, and Spark refine integration.

Testing

• mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest=VectorSearchBuilderTest#testRawCandidateSearchScoresOnlyCandidateBitmap+testBatchRefineReadsUnionCandidatesOnceAndScoresPerQuery+testVectorSearchRefineFactorReranksIndexCandidates+testRawRefineReadTypeContainsOnlyVectorAndRowId test
• python -m pytest paimon-python/pypaimon/tests/vector_search_filter_test.py -k "raw_candidate_search_scores_only_candidate_bitmap or batch_refine_factor_reranks_each_query or refine_factor_reranks_index_candidates_with_raw_vectors or refine_factor_one_reranks_without_expanding_candidates or raw_search_uses_partition_filter_and_index_type_metric"
• mvn -pl paimon-spark/paimon-spark-common -am -Pfast-build -Pspark3 -DfailIfNoTests=false -DwildcardSuites=none -Dtest=SparkVectorReadImplTest#testDistributedIndexRefinesAfterGlobalMerge test
• git diff --check

Notes

The deprecated Java refine overload is kept for compatibility, but internal refine calls now use the bitmap-first path.

[leaves12138]

Looks good to me. I reviewed the vector refine raw-read optimization and the Java/Python/Spark consistency, and the targeted tests passed.