[SPARK-36861][SQL] Use yyyy-MM-dd as the date pattern in partition discovery

[MaxGekk]

What changes were proposed in this pull request?

In the PR, I propose to explicitly set the date pattern to yyyy-MM-dd while inferring types of partition values.

Why are the changes needed?

The existing date partition parser is much more tolerant to its input, and can skip some parts of date strings. For example, see SPARK-36861. As a consequence, it can loose some user's info (pieces of partition values).

Does this PR introduce any user-facing change?

No. New behaviour introduced by #33709 hasn't released yet.

How was this patch tested?

By running the modified test suite:

$ build/sbt "test:testOnly *ParquetV2PartitionDiscoverySuite"
$ build/sbt "test:testOnly *ImageFileFormatSuite"

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50062/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50062/

[SparkQA]

Test build #145590 has finished for PR 34700 at commit c80127c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MaxGekk]

I reverted changes made by https://github.com/apache/spark/pull/33709/files#r688851936. Now the test looks the same as in branch-3.2.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50090/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50090/

[cloud-fan]

thanks, merging to master!

[SparkQA]

Test build #145618 has finished for PR 34700 at commit 505f8ff.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.