Skip to content

[AURON #2172] Add parquet suite wrappers for Spark versions#2281

Open
myandpr wants to merge 5 commits into
apache:masterfrom
myandpr:add-parquet-suite-wrappers
Open

[AURON #2172] Add parquet suite wrappers for Spark versions#2281
myandpr wants to merge 5 commits into
apache:masterfrom
myandpr:add-parquet-suite-wrappers

Conversation

@myandpr
Copy link
Copy Markdown
Contributor

@myandpr myandpr commented May 19, 2026 •
edited
Loading

Which issue does this PR close?

Closes #2172

Rationale for this change

Spark 3.3 already has Parquet correctness test wrappers. This PR extends the same Parquet correctness coverage to the other supported Spark test modules requested by the issue.

What changes are included in this PR?

  • Add Parquet correctness suite wrappers for:
    • Spark 3.1
    • Spark 3.2
    • Spark 3.4
    • Spark 3.5
    • Spark 4.0
    • Spark 4.1
  • Enable the Parquet suites in each corresponding AuronSparkTestSettings.
  • Enable Spark correctness tests in CI for the affected Spark versions.
  • Add version-specific exclusions for unsupported or failing Parquet correctness cases.
  • Skip suites that are not available in the corresponding Spark version.

Are there any user-facing changes?

No. This PR only adds test coverage.

How was this patch tested?

  • Ran affected Spark correctness tests locally.
  • Verified GitHub CI is green.

@myandpr myandpr mentioned this pull request May 20, 2026
Open
6 tasks
@slfan1989 slfan1989 requested a review from Copilot May 20, 2026 15:42
Copilot AI reviewed May 20, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Extends Auron’s Spark Parquet correctness coverage beyond the existing Spark 3.3 module by adding version-specific wrapper suites and enabling them in each Spark test module’s AuronSparkTestSettings, while omitting suites that do not exist in older Spark versions.

Changes:

  • Added AuronParquet* wrapper test suites for Spark 3.1, 3.2, 3.4, 3.5, 4.0, and 4.1 test modules.
  • Enabled the new Parquet wrapper suites in each module’s AuronSparkTestSettings.
  • Version-gated suite coverage by only adding/enabling wrappers for suites present in that Spark version.

Reviewed changes

Copilot reviewed 188 out of 188 changed files in this pull request and generated no comments.

Show a summary per file
File Description
auron-spark-tests/spark31/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala Import parquet package + enable Spark 3.1 Parquet wrapper suites
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala Spark 3.1 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala Import parquet package + enable Spark 3.2 Parquet wrapper suites
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala Spark 3.2 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala Import parquet package + enable Spark 3.4 Parquet wrapper suites
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdIOSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdSchemaSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetVectorizedSuite.scala Spark 3.4 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala Import parquet package + enable Spark 3.5 Parquet wrapper suites
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdIOSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdSchemaSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetVectorizedSuite.scala Spark 3.5 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala Import parquet package + enable Spark 4.0 Parquet wrapper suites
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdIOSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdSchemaSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetVectorizedSuite.scala Spark 4.0 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala Import parquet package + enable Spark 4.1 Parquet wrapper suites
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdIOSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdSchemaSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala Spark 4.1 Parquet suite wrapper
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetVectorizedSuite.scala Spark 4.1 Parquet suite wrapper

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ShreyeshArangath
ShreyeshArangath requested changes May 20, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@ShreyeshArangath ShreyeshArangath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@myandpr I think the tests right now are not running, before we merge more PRs do you mind enabling the correctness tests for the other Spark versions?

  1. auron-spark-tests/pom.xml needs six new profiles (spark-3.1, spark-3.2, spark-3.4, spark-3.5, spark-4.0, spark-4.1) each wiring their sparkXX submodule, mirroring the existing spark-3.3 profile.
  2. github/workflows/tpcds.yml needs sparktests: true added to the test-spark-31, test-spark-32, test-spark-34, test-spark-35-, test-spark-40-, test-spark-41-* jobs. Currently only test-spark-33 has it

You can file a new issue if you'd like

@myandpr
Copy link
Copy Markdown
Contributor Author

myandpr commented May 21, 2026 •
edited
Loading

@myandpr I think the tests right now are not running, before we merge more PRs do you mind enabling the correctness tests for the other Spark versions?

  1. auron-spark-tests/pom.xml needs six new profiles (spark-3.1, spark-3.2, spark-3.4, spark-3.5, spark-4.0, spark-4.1) each wiring their sparkXX submodule, mirroring the existing spark-3.3 profile.
  2. github/workflows/tpcds.yml needs sparktests: true added to the test-spark-31, test-spark-32, test-spark-34, test-spark-35-, test-spark-40-, test-spark-41-* jobs. Currently only test-spark-33 has it

You can file a new issue if you'd like

Hi @ShreyeshArangath Thanks for pointing this out.

Updated the TPC-DS workflow to enable spark correctness tests for the affected Spark versions. The spark test profiles already exist in auron-spark-tests/pom.xml.

Could you please take another look? Thanks!

@github-actions github-actions Bot added the infra label May 21, 2026
@ShreyeshArangath
Copy link
Copy Markdown
Contributor

ShreyeshArangath commented May 21, 2026

Thanks for fixing it! Change looks fine, do you mind looking at the failures?

@github-actions github-actions Bot added the build label May 21, 2026
@myandpr myandpr force-pushed the add-parquet-suite-wrappers branch 2 times, most recently from 14874aa to 110425d Compare May 22, 2026 14:12
@myandpr myandpr force-pushed the add-parquet-suite-wrappers branch from 110425d to 5745ba5 Compare May 22, 2026 14:24
@myandpr myandpr mentioned this pull request May 22, 2026
Closed
@myandpr
Copy link
Copy Markdown
Contributor Author

myandpr commented May 22, 2026

Thanks for fixing it! Change looks fine, do you mind looking at the failures?

@ShreyeshArangath Thanks. The failures have been fixed and CI is green now. Could you please take another look?

@myandpr myandpr requested a review from ShreyeshArangath May 22, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@slfan1989 slfan1989 slfan1989 approved these changes

@ShreyeshArangath ShreyeshArangath Awaiting requested review from ShreyeshArangath

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

build infra spark-tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Correctness Testing] All Spark Versions - Add Parquet related tests

4 participants

@myandpr @ShreyeshArangath @slfan1989