[VL] Fallback Parquet scan if legacy timezone is found in file metadata

[zhztheplayer]

This is to fix result mismatch issues when Gluten is reading Parquet file written by old Spark versions (< 3.0.0) or with legacy datetime rebase mode (spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY / spark.sql.parquet.int96RebaseModeInRead=LEGACY).

Test case Column DEFAULT value support with Delta Lake, positive tests which is being added in #11107 will cover this fix.

Before:

"000[2-12-30]" did not equal "000[1-01-01]"
ScalaTestFailureLocation: org.apache.spark.sql.delta.DeltaColumnDefaultsInsertSuite at (DeltaInsertIntoTableSuite.scala:829)
Expected :"000[1-01-01]"
Actual   :"000[2-12-30]"

After:

15:22:23.853 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: Scan parquet spark_catalog.default.t4[QueryId=43], due to:
        - Detected unsupported metadata in parquet files: Legacy timezone found.

[github-actions]

Run Gluten Clickhouse CI on x86

[FelixYBW]

Is it because that velox doesn't support it?

[zhztheplayer]

Is it because that velox doesn't support it?

Yes, Velox doesn't support the settings.

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[zhztheplayer]

cc @YannByron @FelixYBW

[github-actions]

Run Gluten Clickhouse CI on x86

[Yohahaha]

we should add switch to control it and respect fileLimit, this check is very heavy.

do you test it in tpc benchmark with remote object storage? in my case, tpcds q1 slow down by 10x.

[jinchengchenghh]

The isEncryptionValidationEnabled is false, this change will cause though isEncryptionValidationEnabled is false, fetch all the file, the hadoop listStatus is is heavy

[jinchengchenghh]

Could we detect only one parquet file to read the metadata? If you would like to do the optimization, could you please wait for this PR https://github.com/apache/incubator-gluten/pull/11225/files#diff-169ab07f3b1741a5742eecd22865d93fb00b32028ea4e6dfbeaae2be056a1103R147 to avoid too much conflict?

[FelixYBW]

@zhztheplayer is one parquet file enough or should we sample several files? I remember anohter PR added a sample files number. We may use the same one.

[jinchengchenghh]

Now the fileLimit is a new config default 10, but rootpaths is Seq, so it is 10 * rootpaths.length

[zhztheplayer]

The PR still respects the previous sample number. There should be another bug. I'll get back to this.

[zhztheplayer]

Could we detect only one parquet file to read the metadata? If you would like to do the optimization, could you please wait for this PR https://github.com/apache/incubator-gluten/pull/11225/files#diff-169ab07f3b1741a5742eecd22865d93fb00b32028ea4e6dfbeaae2be056a1103R147 to avoid too much conflict?

@jinchengchenghh Let's first address the performance regressions as it affects all main stream code users. I opened a PR #11233.

[zhztheplayer]

do you test it in tpc benchmark with remote object storage? in my case, tpcds q1 slow down by 10x.

No I missed out the part that the encryption validation was disabled by default. I think PR #11233 should temporarily fix the problem.

If Q1 is slowed down that much by footer reading, and provided Spark also needs to read the footers to infer Parquet schema, perhaps it might be beneficial if we follow Spark's practice to parallelize the footer reading in the future:

https://github.com/apache/spark/blob/54cde812c2657717651156636041a84556619a5b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L539-L561