[SPARK-32909][SQL] Pass all sql/hive-thriftserver module UTs in Scala 2.13

[LuciferYang]

What changes were proposed in this pull request?

This pr fix failed and aborted cases in sql hive-thriftserver module in Scala 2.13, the main change of this pr as follow:

• Use s.c.Seq instead of Seq in HiveResult because the input type maybe mutable.ArraySeq, but Seq represent immutable.Seq in Scala 2.13.

• Reset classLoader after HiveMetastoreLazyInitializationSuite completed because context class loader is NonClosableMutableURLClassLoader in HiveMetastoreLazyInitializationSuite running process, and it propagate to HiveThriftServer2ListenerSuite trigger following problems in Scala 2.13:

HiveThriftServer2ListenerSuite:
*** RUN ABORTED ***
  java.lang.LinkageError: loader constraint violation: loader (instance of net/bytebuddy/dynamic/loading/MultipleParentClassLoader) previously initiated loading for a different type with name "org/apache/hive/service/ServiceStateChangeListener"
  at org.mockito.codegen.HiveThriftServer2$MockitoMock$1850222569.<clinit>(Unknown Source)
  at sun.reflect.GeneratedSerializationConstructorAccessor530.newInstance(Unknown Source)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:48)
  at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:73)
  at org.mockito.internal.creation.instance.ObjenesisInstantiator.newInstance(ObjenesisInstantiator.java:19)
  at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMock(SubclassByteBuddyMockMaker.java:47)
  at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:25)
  at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:35)
  at org.mockito.internal.MockitoCore.mock(MockitoCore.java:63)
  ...

After this pr HiveThriftServer2Suites and HiveThriftServer2ListenerSuite was fixed and all 461 test passed

Why are the changes needed?

We need to support a Scala 2.13 build.

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Scala 2.12: Pass the Jenkins or GitHub Action

• Scala 2.13: All tests passed.

Do the following:

dev/change-scala-version.sh 2.13
mvn clean install -DskipTests -pl sql/hive-thriftserver -am -Phive-thriftserver -Pscala-2.13
mvn test -pl sql/hive-thriftserver -Phive -Phive-thriftserver -Pscala-2.13

Before

HiveThriftServer2ListenerSuite:
*** RUN ABORTED ***

After

Tests: succeeded 461, failed 0, canceled 0, ignored 17, pending 0
All tests passed.

[LuciferYang]

cc @srowen I think this change is very Interesting, HiveThriftServer2ListenerSuite will ABORTED without this change

HiveThriftServer2ListenerSuite:
*** RUN ABORTED ***
  java.lang.LinkageError: loader constraint violation: loader (instance of net/bytebuddy/dynamic/loading/MultipleParentClassLoader) previously initiated loading for a different type with name "org/apache/hive/service/ServiceStateChangeListener"

The Context Class Loader is NonClosableMutableURLClassLoader in HiveMetastoreLazyInitializationSuite running process and propagate to HiveThriftServer2ListenerSuite , then

spark/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2ListenerSuite.scala

Lines 172 to 175 in 92b75dc

	if (live) {
	val server = mock(classOf[HiveThriftServer2], RETURNS_SMART_NULLS)
	val listener = new HiveThriftServer2Listener(kvstore, sparkConf, Some(server))
	(new HiveThriftServer2AppStatusStore(kvstore, Some(listener)), listener)

will use MultipleParentClassLoader(NonClosableMutableURLClassLoader and AppClassLoader) to mock HiveThriftServer2 instance in Scala 2.13, but mock HiveThriftServer2 instance use AppClassLoader only in Scala 2.12.

Then I do some experiments:

• Ignore HiveMetastoreLazyInitializationSuite all tests passed

• Run HiveThriftServer2ListenerSuite only will passed

• Copy test case HiveMetastoreLazyInitializationSuite to HiveThriftServer2ListenerSuite then run HiveThriftServer2ListenerSuite only, it aborted in both Scala 2.12 and Scala 2.13.

I guess that HiveMetastoreLazyInitializationSuite and HiveThriftServer2ListenerSuite run in same thread in Scala 2.13 but different thread in Scala 2.12, but I don't know why.

[dongjoon-hyun]

Thank you for investigation, @LuciferYang . It's interesting.

[srowen]

Interesting. I think it's fine, as this change just improves isolation in any event.

[dongjoon-hyun]

Ya. I agree with @srowen . The patch itself looks good.

[SparkQA]

Test build #128811 has finished for PR 29783 at commit 0d20878.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Retest this please

[dongjoon-hyun]

I triggered JDK11 testing too because this is a class loader related issue, @LuciferYang .

[SparkQA]

Test build #128831 has finished for PR 29783 at commit 0d20878.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Thank you, @LuciferYang and @srowen .
Merged to master for Apache Spark 3.1.0 on December 2020.

[LuciferYang]

Thank @srowen and @dongjoon-hyun ~