[SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown

[goutamadwant]

What changes were proposed in this pull request?

Run SchemaPruning after PushVariantIntoScan in the early scan pushdown batch. This removes variant reconstruction projections that become unnecessary during variant pushdown and allows the Once batch to reach the same plan on its first application.

Add a Parquet V1 regression test covering a query that reads _metadata.file_path while a VARIANT column is referenced below a nondeterministic filter.

Closes #57659.

Why are the changes needed?

PushVariantIntoScan can make a variant reconstruction projection unnecessary after the earlier schema-pruning passes have completed. Reapplying the optimizer batch then removes that projection, which violates the idempotence requirement for a Once batch and causes the reported query to fail during optimization.

Running SchemaPruning once after variant pushdown makes the first application reach the stable plan.

Does this PR introduce any user-facing change?

Yes. Queries that select file metadata while pruning a VARIANT column below a nondeterministic filter no longer fail during optimization with a Once batch idempotence error.

How was this patch tested?

The new regression test was confirmed to fail before the optimizer change and pass afterward.

The following tests and checks passed:

• ./build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetV1SchemaPruningSuite -- -z "SPARK-57659"'
• ./build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetV1SchemaPruningSuite org.apache.spark.sql.execution.datasources.PushVariantIntoScanSuite org.apache.spark.sql.execution.datasources.PushVariantIntoScanVectorizedSuite' (349 tests)
• ./build/sbt 'sql/scalastyle' 'sql/Test/scalastyle'
• git diff --check

The full ./dev/run-tests suite was not run locally.

Was this patch authored or co-authored using generative AI tooling?

AI was used to review the code and understand the existing codebase.

This contribution is my original work, and I license it under the project's open source license.

[felipepessoto]

@uros-b , @goutamadwant any idea why CI is not running?

[uros-b]

@goutamadwant @felipepessoto Please try to follow the instructions in https://github.com/apache/spark/pull/57809/checks?check_run_id=92506854083.

[uros-b]

For more information, please see section "Pull request" in https://spark.apache.org/contributing.html

[goutamadwant]

@uros-b , @goutamadwant any idea why CI is not running?

@uros-b @felipepessoto Thanks for the links. I enabled the fork workflows and followed the steps. I see upstream build check has been detected

[felipepessoto]

The docker integration test failed with an error that seems unrelated:

[info] Cause: oracle.net.ns.NetException: ORA-12516: Cannot connect to database. Listener at host 10.1.0.191 port 32769 does not have a protocol handler for TCP ready or registered for service freepdb1.

@uros-b can we ignore and merge it, or do we need to re-run?