[SPARK-31414][SQL] Fix performance regression with new TimestampFormatter for json and csv time parsing

[yaooqinn]

What changes were proposed in this pull request?

With benchmark original, where the timestamp values are valid to the new parser

the result is

[info] Running benchmark: Read dates and timestamps
[info]   Running case: timestamp strings
[info]   Stopped after 3 iterations, 5781 ms
[info]   Running case: parse timestamps from Dataset[String]
[info]   Stopped after 3 iterations, 44764 ms
[info]   Running case: infer timestamps from Dataset[String]
[info]   Stopped after 3 iterations, 93764 ms
[info]   Running case: from_json(timestamp)
[info]   Stopped after 3 iterations, 59021 ms

When we modify the benchmark to

     def timestampStr: Dataset[String] = {
        spark.range(0, rowsNum, 1, 1).mapPartitions { iter =>
          iter.map(i => s"""{"timestamp":"1970-01-01T01:02:03.${i % 100}"}""")
        }.select($"value".as("timestamp")).as[String]
      }

      readBench.addCase("timestamp strings", numIters) { _ =>
        timestampStr.noop()
      }

      readBench.addCase("parse timestamps from Dataset[String]", numIters) { _ =>
        spark.read.schema(tsSchema).json(timestampStr).noop()
      }

      readBench.addCase("infer timestamps from Dataset[String]", numIters) { _ =>
        spark.read.json(timestampStr).noop()
      }

where the timestamp values are invalid for the new parser which causes a fallback to legacy parser(2.4).
the result is

[info] Running benchmark: Read dates and timestamps
[info]   Running case: timestamp strings
[info]   Stopped after 3 iterations, 5623 ms
[info]   Running case: parse timestamps from Dataset[String]
[info]   Stopped after 3 iterations, 506637 ms
[info]   Running case: infer timestamps from Dataset[String]
[info]   Stopped after 3 iterations, 509076 ms

About 10x perf-regression

BUT if we modify the timestamp pattern to ....HH:mm:ss[.SSS][XXX] which make all timestamp values valid for the new parser to prohibit fallback, the result is

[info] Running benchmark: Read dates and timestamps
[info]   Running case: timestamp strings
[info]   Stopped after 3 iterations, 5623 ms
[info]   Running case: parse timestamps from Dataset[String]
[info]   Stopped after 3 iterations, 506637 ms
[info]   Running case: infer timestamps from Dataset[String]
[info]   Stopped after 3 iterations, 509076 ms

Why are the changes needed?

Fix performance regression.

Does this PR introduce any user-facing change?

NO

How was this patch tested?

new tests added.

[SparkQA]

Test build #121087 has finished for PR 28181 at commit 4e835fd.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #121091 has finished for PR 28181 at commit cc0d9a0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #121094 has finished for PR 28181 at commit ca902b2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #121095 has finished for PR 28181 at commit a63824b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

cc @MaxGekk

[yaooqinn]

cc: @cloud-fan @maropu @dongjoon-hyun, thanks

[cloud-fan]

good catch! LGTM, merging to master/3.0