[SPARK-28158][SQL] Hive UDFs supports UDT type

[uncleGen]

What changes were proposed in this pull request?

After this PR, we can create and register Hive UDFs to accept UDT type, like VectorUDT and MatrixUDT. These UDTs are widely used in Spark machine learning.

How was this patch tested?

add new ut

[SparkQA]

Test build #106870 has finished for PR 24961 at commit 9039549.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[uncleGen]

retest this please.

[SparkQA]

Test build #106882 has finished for PR 24961 at commit 9039549.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[uncleGen]

@cloud-fan Could you please help to review?

[maropu]

How do you handle these type data in Hive UDFs?

[uncleGen]

@maropu For example, the internal sql type of VectorUDT and MatrixUDT is StructType. We can use StructObjectInspector in Hive UDFs to handle there type data. There is a demo: https://github.com/aliyun/aliyun-emapreduce-sdk/blob/master-2.x/emr-sql/src/main/scala/org/apache/spark/sql/aliyun/udfs/ml/LogisticRegressionUDF.scala#L86

[maropu]

But, the UDF works in Spark and not in Hive, right? I'm not sure that this is a right approach...

[uncleGen]

@maropu I dont test the UDF in Hive, but I think it will work as it is a Hive UDF definition.

[cloud-fan]

Can we add an end-to-end test to show the value of this patch?

[SparkQA]

Test build #107684 has finished for PR 24961 at commit 3faaa00.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class VectorUDTSuite extends QueryTest

[SparkQA]

Test build #107685 has finished for PR 24961 at commit 9cadbe4.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class VectorUDTSuite extends QueryTest

[HyukjinKwon]

ping @uncleGen

[uncleGen]

let me continue to finish this pr

[SparkQA]

Test build #111339 has finished for PR 24961 at commit a8e2fb3.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class VectorUDTSuite extends SparkFunSuite
• class HiveUserDefinedTypeSuite extends QueryTest
• class TestUDF extends GenericUDF

[SparkQA]

Test build #111340 has finished for PR 24961 at commit 7138d6a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class VectorUDTSuite extends SparkFunSuite
• class HiveUserDefinedTypeSuite extends QueryTest
• class TestUDF extends GenericUDF

[SparkQA]

Test build #111338 has finished for PR 24961 at commit 12ee9d6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class VectorUDTSuite extends QueryTest

[SparkQA]

Test build #111348 has finished for PR 24961 at commit 4a4e75c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[uncleGen]

ping @HyukjinKwon @cloud-fan

[SparkQA]

Test build #112603 has finished for PR 24961 at commit b71273d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #112757 has finished for PR 24961 at commit ab0282e.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class HiveUserDefinedTypeSuite extends QueryTest with TestHiveSingleton

[HyukjinKwon]

retest this please

[SparkQA]

Test build #112762 has finished for PR 24961 at commit ab0282e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class HiveUserDefinedTypeSuite extends QueryTest with TestHiveSingleton

[HyukjinKwon]

Merged to master.

[HeartSaVioR]

I've submitted a follow-up PR as new test is high likely failing in recent CI builds. #26287