[SPARK-34772][SQL] RebaseDateTime loadRebaseRecords should use Spark classloader instead of context

[ulysses-you]

What changes were proposed in this pull request?

Change context classloader to Spark classloader at RebaseDateTime.loadRebaseRecords

Why are the changes needed?

With custom spark.sql.hive.metastore.version and spark.sql.hive.metastore.jars.

Spark would use date formatter in HiveShim that convert date to string, if we set spark.sql.legacy.timeParserPolicy=LEGACY and the partition type is date the RebaseDateTime code will be invoked. At that moment, if RebaseDateTime is initialized the first time then context class loader is IsolatedClientLoader. Such error msg would throw:

java.lang.IllegalArgumentException: argument "src" is null
  at com.fasterxml.jackson.databind.ObjectMapper._assertNotNull(ObjectMapper.java:4413)
  at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3157)
  at com.fasterxml.jackson.module.scala.ScalaObjectMapper.readValue(ScalaObjectMapper.scala:187)
  at com.fasterxml.jackson.module.scala.ScalaObjectMapper.readValue$(ScalaObjectMapper.scala:186)
  at org.apache.spark.sql.catalyst.util.RebaseDateTime$$anon$1.readValue(RebaseDateTime.scala:267)
  at org.apache.spark.sql.catalyst.util.RebaseDateTime$.loadRebaseRecords(RebaseDateTime.scala:269)
  at org.apache.spark.sql.catalyst.util.RebaseDateTime$.<init>(RebaseDateTime.scala:291)
  at org.apache.spark.sql.catalyst.util.RebaseDateTime$.<clinit>(RebaseDateTime.scala)
  at org.apache.spark.sql.catalyst.util.DateTimeUtils$.toJavaDate(DateTimeUtils.scala:109)
  at org.apache.spark.sql.catalyst.util.LegacyDateFormatter.format(DateFormatter.scala:95)
  at org.apache.spark.sql.catalyst.util.LegacyDateFormatter.format$(DateFormatter.scala:94)
  at org.apache.spark.sql.catalyst.util.LegacySimpleDateFormatter.format(DateFormatter.scala:138)
  at org.apache.spark.sql.hive.client.Shim_v0_13$ExtractableLiteral$1$.unapply(HiveShim.scala:661)
  at org.apache.spark.sql.hive.client.Shim_v0_13.convert$1(HiveShim.scala:785)
  at org.apache.spark.sql.hive.client.Shim_v0_13.$anonfun$convertFilters$4(HiveShim.scala:826)

java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.sql.catalyst.util.RebaseDateTime$
  at org.apache.spark.sql.catalyst.util.DateTimeUtils$.toJavaDate(DateTimeUtils.scala:109)
  at org.apache.spark.sql.catalyst.util.LegacyDateFormatter.format(DateFormatter.scala:95)
  at org.apache.spark.sql.catalyst.util.LegacyDateFormatter.format$(DateFormatter.scala:94)
  at org.apache.spark.sql.catalyst.util.LegacySimpleDateFormatter.format(DateFormatter.scala:138)
  at org.apache.spark.sql.hive.client.Shim_v0_13$ExtractableLiteral$1$.unapply(HiveShim.scala:661)
  at org.apache.spark.sql.hive.client.Shim_v0_13.convert$1(HiveShim.scala:785)
  at org.apache.spark.sql.hive.client.Shim_v0_13.$anonfun$convertFilters$4(HiveShim.scala:826)
  at scala.collection.immutable.Stream.flatMap(Stream.scala:493)
  at org.apache.spark.sql.hive.client.Shim_v0_13.convertFilters(HiveShim.scala:826)
  at org.apache.spark.sql.hive.client.Shim_v0_13.getPartitionsByFilter(HiveShim.scala:848)
  at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getPartitionsByFilter$1(HiveClientImpl.scala:749)
  at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:291)
  at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:224)
  at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:223)
  at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:273)
  at org.apache.spark.sql.hive.client.HiveClientImpl.getPartitionsByFilter(HiveClientImpl.scala:747)
  at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$listPartitionsByFilter$1(HiveExternalCatalog.scala:1273)

The reproduce steps:

1. spark.sql.hive.metastore.version and spark.sql.hive.metastore.jars.
2. CREATE TABLE t (c int) PARTITIONED BY (p date)
3. SET spark.sql.legacy.timeParserPolicy=LEGACY
4. SELECT * FROM t WHERE p='2021-01-01'

Does this PR introduce any user-facing change?

Yes, bug fix.

How was this patch tested?

pass org.apache.spark.sql.catalyst.util.RebaseDateTimeSuite and add new unit test to HiveSparkSubmitSuite.scala.

[yaooqinn]

Seems the *-rebase-micros.json can not be reached in the isolated classloader for hive client. cc @MaxGekk @cloud-fan

[cloud-fan]

@ulysses-you thanks for catching this bug! do you know the steps to reproduce this bug manually? so that other people can try it out.

[cloud-fan]

also cc @wangyum

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40728/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40728/

[ulysses-you]

The condition is a little hard, has updated at the description and also add an other unknown error msg. @cloud-fan @wangyum

[wangyum]

Thank you @ulysses-you I can reproduce it:

bin/spark-sql --conf spark.sql.hive.metastore.version=1.2.1 --conf spark.sql.legacy.timeParserPolicy=LEGACY --conf spark.sql.hive.metastore.jars=maven

CREATE TABLE t (c int) PARTITIONED BY (p date);
SELECT * FROM t WHERE p='2021-01-01';

Could we add a test to HiveSparkSubmitSuite?

[wangyum]

Please avoid use hardcoded configs:

"--conf", s"${HiveUtils.HIVE_METASTORE_VERSION.key}=1.2.1",
"--conf", s"${HiveUtils.HIVE_METASTORE_JARS.key}=maven",
"--conf", s"${SQLConf.LEGACY_TIME_PARSER_POLICY.key}=${LegacyBehaviorPolicy.LEGACY.toString}",

[wangyum]

Disable UI?

.config(UI_ENABLED.key, "false")

[wangyum]

.collect() is better than .show().

[yaooqinn]

nit: need ->need to..... version ->versions.

[HyukjinKwon]

The main code change looks fine.

[cloud-fan]

LGTM. How far shall we backport this? 3.0?

[ulysses-you]

RebaseDateTime is created in branch-3.0, so it would be better to backport 3.0.

[wangyum]

Yes it is. It is better to backport it to branch-3.0.

[ulysses-you]

thanks all !

[wangyum]

Merged to master, branch-3.1 and branch-3.0.

[dongjoon-hyun]

Hi, All.
The test case seems to be broken in Java 11 environment. Could you take a look at that?

$ build/sbt "hive/testOnly *.HiveSparkSubmitSuite -- -z SPARK-34772" -Phive
...
[info] *** 1 TEST FAILED ***
[error] Failed: Total 1, Failed 1, Errors 0, Passed 0
[error] Failed tests:
[error] 	org.apache.spark.sql.hive.HiveSparkSubmitSuite
[error] (hive / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 328 s (05:28), completed Mar 21, 2021, 5:32:39 PM

[dongjoon-hyun]

I made a follow-up PR.

• [SPARK-34772][TESTS][FOLLOWUP] Disable a test case using Hive 1.2.1 in Java9+ environment #31916