Pluggable Scan Planning #2303
Description
Feature Request / Improvement
Umbrella issue. Today we have tightly coupled query planning in Python. This works well but runs into limitations when it comes to concurrency and performance. Having pluggable query planning will allow for:
- Python based query planning which is working today and is proven.
- Iceberg-Rust based query planning as suggeted in Upserting large table extremely slow #2159 (comment).
- REST Catalog server side planning this leverages the REST Catalog to plan the query, rather than the client. First issue can be found here: REST: Add objects for server side planning #2302.
As an API, I think it makes sense to follow the Server Side planning. So a PlanTableScanRequest as an input, and the ScanTasks as the output.