Sub-issue of #4098.
Description
Two tests fail in both `Spark 4.1, JDK 17/auto [parquet]` (Linux) and `macos-14/Spark 4.1, JDK 17, Scala 2.13 [parquet]`:
- `native reader - select struct field with user defined schema - native_datafusion`
- `native reader - select struct field with user defined schema - native_iceberg_compat`
Symptom: `Results do not match for query`. The schema involved is `c0: struct<y:int,x:string>` over a parquet relation. Comet's native reader returns different rows than Spark.
Suspected root cause
Spark 4.1 likely changed how user-supplied struct schemas are reconciled with on-disk Parquet field order, or field pruning behaves differently. Compare Spark 4.0 vs 4.1 planning output for this query and check whether user-schema field-name-vs-position behavior changed in `ParquetReadSupport` or `ParquetSchemaConverter`.
Where
The test currently has `assume(!isSpark41Plus, "https://github.com/apache/datafusion-comet/issues/4098")\` in `spark/src/test/scala/org/apache/comet/exec/CometNativeReaderSuite.scala` (test name "native reader - select struct field with user defined schema").
Sub-issue of #4098.
Description
Two tests fail in both `Spark 4.1, JDK 17/auto [parquet]` (Linux) and `macos-14/Spark 4.1, JDK 17, Scala 2.13 [parquet]`:
Symptom: `Results do not match for query`. The schema involved is `c0: struct<y:int,x:string>` over a parquet relation. Comet's native reader returns different rows than Spark.
Suspected root cause
Spark 4.1 likely changed how user-supplied struct schemas are reconciled with on-disk Parquet field order, or field pruning behaves differently. Compare Spark 4.0 vs 4.1 planning output for this query and check whether user-schema field-name-vs-position behavior changed in `ParquetReadSupport` or `ParquetSchemaConverter`.
Where
The test currently has `assume(!isSpark41Plus, "https://github.com/apache/datafusion-comet/issues/4098")\` in `spark/src/test/scala/org/apache/comet/exec/CometNativeReaderSuite.scala` (test name "native reader - select struct field with user defined schema").