Motivation
paimon_datafusion::SQLContext currently creates a DataFusion session with the default RuntimeEnv. Applications embedding Paimon Rust therefore cannot provide a bounded execution memory pool, configure a spill-aware pool, select a temporary directory, or cap temporary disk usage.
Long-running services and other resource-constrained applications need to control these DataFusion runtime resources without replacing Paimon's SQL context and its custom planners, optimizer rules, catalogs, and functions.
Proposal
- Add a backward-compatible builder for
SQLContext.
- Allow Rust callers to provide a custom DataFusion
RuntimeEnv while retaining all Paimon-specific session configuration.
- Expose optional, generic memory-pool and temporary-directory settings from the Python
SQLContext binding.
- Keep
SQLContext::new() and zero-argument Python SQLContext() behavior unchanged.
Non-goals
- Detecting container or host resource limits.
- Managing application-specific temporary-directory lifecycle.
- Guaranteeing spill support for operators that DataFusion itself cannot spill.
- Accounting for allocations outside DataFusion's memory pool.
Motivation
paimon_datafusion::SQLContextcurrently creates a DataFusion session with the defaultRuntimeEnv. Applications embedding Paimon Rust therefore cannot provide a bounded execution memory pool, configure a spill-aware pool, select a temporary directory, or cap temporary disk usage.Long-running services and other resource-constrained applications need to control these DataFusion runtime resources without replacing Paimon's SQL context and its custom planners, optimizer rules, catalogs, and functions.
Proposal
SQLContext.RuntimeEnvwhile retaining all Paimon-specific session configuration.SQLContextbinding.SQLContext::new()and zero-argument PythonSQLContext()behavior unchanged.Non-goals