feat(rust/sedona): Auto-configure spilled batch in-memory size threshold based on global memory limit

[Kontinuation]

Summary

• Automatically configure the spilled_batch_in_memory_size_threshold based on the global memory limit, setting it to 5% of the per-partition memory limit (with a minimum of 10MB).
• This is a by-product of writing documentation for configuring memory management and spilling (docs: add memory management and spill configuration guide #679). We decided to configure this option automatically instead of requiring users to set it up manually based on the global memory limit.

Tests

We've tested running spatial join queries in SpatialBench SF=10 and 100, and also some spatial joins involving OvertureMaps datasets (address, building, landuse), and found that this default value worked pretty well. Users can still override the default value using SET sedona.spatial_join.spilled_batch_in_memory_size_threshold = <custom_value>.

[Copilot]

Pull request overview

Automatically derives Sedona’s spatial-join spill batching threshold from the configured DataFusion global memory limit, to reduce manual tuning when enabling spilling/memory limits.

Changes:

• Reads RuntimeEnv memory pool limit and computes a per-partition limit using target_partitions.
• Sets sedona.spatial_join.spilled_batch_in_memory_size_threshold to 5% of per-partition memory (min 10MB) when the memory limit is finite.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[paleolimbot]

Thank you!

[paleolimbot]

Merging so I can do a few tests!