chore(rust/sedona-spatial-join): Evaluate spatial predicate operands in EvaluateOperandBatchStream

[Kontinuation]

This refactor makes BuildSideBatchesCollector and SpatialJoinStream work with streams of EvaluatedBatches instead of directly with streams of RecordBatches. We will have EvaluateBatches read directly by spill readers so adding this layer of abstraction make the main part of spatial join care less about whether the stream is directly from the source of read from spill files.

The stream for the build-side automatically compact batches to avoid holding large sparse binary view arrays in memory. The EvaluateOperandBatchStream performs the batch compaction automatically before evaluating the operands of spatial predicates.

[Copilot]

Pull request overview

This PR refactors the spatial join implementation to work with EvaluatedBatch streams instead of RecordBatch streams. The changes introduce a new EvaluateOperandBatchStream that evaluates spatial predicate operands and automatically compacts batches to optimize memory usage.

Changes:

• Introduces compact_batch and compact_array functions to reorganize payload buffers in view arrays
• Creates EvaluateOperandBatchStream to evaluate geometry expressions and produce EvaluatedBatches
• Updates BuildSideBatchesCollector and SpatialJoinStream to consume EvaluatedBatch streams

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
rust/sedona-spatial-join/src/utils/arrow_utils.rs	Adds compaction utilities for view arrays to reduce memory usage
rust/sedona-spatial-join/src/stream.rs	Updates probe stream to use evaluated batches
rust/sedona-spatial-join/src/index/build_side_collector.rs	Refactors to consume evaluated batch streams and adds concurrent/sequential collection modes
rust/sedona-spatial-join/src/evaluated_batch/evaluated_batch_stream/in_mem.rs	Adds schema field to in-memory stream
rust/sedona-spatial-join/src/evaluated_batch/evaluated_batch_stream/evaluate.rs	New file implementing operand evaluation stream
rust/sedona-spatial-join/src/evaluated_batch/evaluated_batch_stream.rs	Adds schema method to trait
rust/sedona-spatial-join/src/build_index.rs	Simplifies partition collection logic

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[paleolimbot]

Thank you!

[paleolimbot]

I don't see a test for this and the below. Perhaps we could test these branches here or remove them and include this code as a suggested future improvement?

[Kontinuation]

I have added several test cases to cover the optimized case.