[SPARK-51661][SQL] Partitions discovery of TIME column values

[MaxGekk]

What changes were proposed in this pull request?

In the PR, I propose to infer the TIME data type from partition values that match to the pattern HH:mm:ss[.SSSSSS]. The second fraction part has variable length namely the following values match to the pattern: 01:02:03.001, 23:59:59 and 12:13:14.123456.

Why are the changes needed?

Currently, Spark can save a dataset partitioned by a TIME column, and read it back if an user set a schema explicitly, but it cannot infer the TIME data type of the column automatically. For example:

scala> sql("SELECT time'12:00' AS t, 0 as id").write.partitionBy("t").parquet("/Users/maxim.gekk/tmp/time_parquet2")
scala> spark.read.parquet("/Users/maxim.gekk/tmp/time_parquet2").printSchema()
root
 |-- id: integer (nullable = true)
 |-- t: string (nullable = true)

Does this PR introduce any user-facing change?

Yes. After the changes, the inferred type is TIME(6) instead of STRING for the example above:

scala> spark.read.parquet("/Users/maxim.gekk/tmp/time_parquet2").printSchema()
root
 |-- id: integer (nullable = true)
 |-- t: time(6) (nullable = true)

How was this patch tested?

By running new test:

$ build/sbt "test:testOnly *ParquetV1PartitionDiscoverySuite"
$ build/sbt "test:testOnly *ParquetV2PartitionDiscoverySuite"

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Thank you for putting this here.

[dongjoon-hyun]

+1, this looks good to me. Thank you, @MaxGekk .

[MaxGekk]

Merging to master. Thank you, @dongjoon-hyun @HyukjinKwon for review.