[AURON #2163] Support native Iceberg scans with residual filters via scan pruning and post-scan native filter

[weimingdiit]

Which issue does this PR close?

Closes #2163

Rationale for this change

The previous behavior was too conservative for Iceberg scans with residual filters. Even when the scan could still be executed natively and the remaining filter logic could be handled above the scan, the planner would fall back entirely.

This PR improves native coverage for Iceberg reads by:

• preserving correctness for unsupported predicates
• increasing native scan applicability for common filter patterns
• reusing the existing native filter path instead of requiring full scan-level predicate support up front

This is an incremental improvement to Iceberg native execution, not full Iceberg feature coverage.

What changes are included in this PR?

This PR:

• removes the unconditional fallback for Iceberg scans with non-alwaysTrue residual filters
• extends IcebergScanPlan to carry pruningPredicates
• extracts Iceberg scan filter expressions and converts a supported subset into Spark expressions
• converts those Spark expressions into native scan pruning predicates
• passes pruning predicates down through NativeIcebergTableScanExec
• keeps unsupported predicates on the upper NativeFilter path
• adds integration coverage for:
  • equality-based pruning
  • IN-based pruning
  • partial pushdown where only part of the predicate is pushed to scan pruning

Supported predicate scope in this PR

The scan-pruning conversion added here supports a limited subset of Iceberg expressions, including:

• AND
• OR
• NOT
• IS NULL
• IS NOT NULL
• IS NAN
• NOT NAN
• comparison predicates such as =, !=, <, <=, >, >=
• IN
• NOT IN

The current implementation intentionally avoids pushing some types through scan pruning, including:

• StringType
• BinaryType
• DecimalType

Unsupported predicates are not pushed into scan pruning and are instead left for post-scan native filtering.

How was this patch tested?

Integration coverage was added in AuronIcebergIntegrationSuite

[Copilot]

Pull request overview

This PR improves native execution coverage for Iceberg table reads by allowing native Iceberg scans to proceed even when Iceberg reports residual filters, and by pushing a supported subset of Iceberg filter expressions into native scan-pruning predicates while leaving unsupported parts to be evaluated via the existing native post-scan filter path.

Changes:

• Extend IcebergScanPlan to carry pruningPredicates derived from Iceberg scan filter expressions.
• Pass pruning predicates into NativeIcebergTableScanExec so native Parquet/ORC scans can apply scan pruning.
• Add/adjust Iceberg integration tests to validate pruning predicate population and partial pushdown behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
thirdparty/auron-iceberg/src/test/scala/org/apache/auron/iceberg/AuronIcebergIntegrationSuite.scala	Adds integration coverage for pruning predicates and partial pushdown scenarios.
thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeIcebergTableScanExec.scala	Threads pruningPredicates into the native scan protobuf nodes.
thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/auron/iceberg/IcebergScanSupport.scala	Removes residual-filter fallback and adds filter-expression extraction + conversion into native scan pruning predicates.
spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala	Ensures rename-column handling treats NativeIcebergTableScan as a scan requiring renames.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[slfan1989]

@weimingdiit Thanks for the contribution! @yew1eb Thanks for the review!

[weimingdiit]

@slfan1989 @yew1eb Thanks for your review and merge