[native_datafusion] [Spark SQL Tests] Dynamic Partition Pruning (DPP) not working correctly

[andygrove]

Summary

3 Spark SQL tests fail due to DPP-related issues when native_datafusion scan is used in auto mode.

Failing Tests

• DynamicPartitionPruningSuite: "static scan metrics" — scanOption.isDefined was false
• ExplainSuite: "explain formatted - check presence of subquery in case of DPP" — expected 2 subqueries, found 0
• Multiple TASK_WRITE_FAILED errors in DPP-related stages

Root Cause

nativeDataFusionScan already has a check for dynamic pruning filters (isDynamicPruningFilter), but these tests use DPP scenarios (e.g. static pruning, subqueries) that may bypass that check. The fallback logic may need to be broadened, or the plan structure produced by native_datafusion may not be compatible with DPP expectations.

Related

Discovered in CI for #3307 (enable native_datafusion in auto scan mode).