[SPARK-6747][SQL] Support List<> as a return type in Hive UDF #5395

maropu · 2015-04-07T17:17:53Z

This patch supports List<> as a return type in Hive UDF.

We assume an UDF below;
public class UDFToListString extends UDF {
public List evaluate(Object o)
{ return Arrays.asList("xxx", "yyy", "zzz"); }
}
An exception of scala.MatchError is thrown as follows when the UDF used in the current implementation.
scala.MatchError: interface java.util.List (of class java.lang.Class)
at org.apache.spark.sql.hive.HiveInspectors$class.javaClassToDataType(HiveInspectors.scala:174)
at org.apache.spark.sql.hive.HiveSimpleUdf.javaClassToDataType(hiveUdfs.scala:76)
at org.apache.spark.sql.hive.HiveSimpleUdf.dataType$lzycompute(hiveUdfs.scala:106)
at org.apache.spark.sql.hive.HiveSimpleUdf.dataType(hiveUdfs.scala:106)
at org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:131)
at org.apache.spark.sql.catalyst.planning.PhysicalOperation$$anonfun$collectAliases$1.applyOrElse(patterns.scala:95)
at org.apache.spark.sql.catalyst.planning.PhysicalOperation$$anonfun$collectAliases$1.applyOrElse(patterns.scala:94)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at scala.collection.TraversableLike$$anonfun$collect$1.apply(TraversableLike.scala:278)
...

AmplabJenkins · 2015-04-07T17:22:11Z

Can one of the admins verify this patch?

chenghao-intel · 2015-04-07T17:34:35Z

sql/hive/src/test/java/org/apache/spark/sql/hive/execution/UDFToListString.java

+    public List<String> evaluate(Object o) {
+        return Arrays.asList("data1", "data2", "data3");
+    }
+}


Add a blank line at the end of file.

chenghao-intel · 2015-04-07T17:40:20Z

@maropu my concern is does Hive support the UDF which return type is List<Object>? Can you confirm that? Or can you provide a Hive comparison unit test?

marmbrus · 2015-04-07T18:24:55Z

ok to test

SparkQA · 2015-04-07T18:27:43Z

Test build #29807 has started for PR 5395 at commit bd165b9.

SparkQA · 2015-04-07T19:56:34Z

Test build #29807 has finished for PR 5395 at commit bd165b9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-07T19:56:38Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29807/
Test PASSed.

SparkQA · 2015-04-08T02:23:21Z

Test build #29825 has started for PR 5395 at commit 02b3a91.

maropu · 2015-04-08T02:31:41Z

Ok, I will look into the implementation and the documentation of Hive for that.

SparkQA · 2015-04-08T03:43:31Z

Test build #29825 has finished for PR 5395 at commit 02b3a91.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-08T03:43:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29825/
Test PASSed.

maropu · 2015-04-11T06:58:05Z

ISTM hive supports list<> as a return type (see the links below).
Also, some thrid-party libraries use it.

https://github.com/kyluka/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java#L163
https://github.com/l1x/apache-hive/blob/master/hive-0.8.1/src/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L113

marmbrus · 2015-04-11T21:45:38Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala

    // Hive seems to return this for struct types?
    case c: Class[_] if c == classOf[java.lang.Object] => NullType
+
+    case c => throw new HiveDataTypeException("Unknown java type: " + c)


This should just be an AnalysisException.

Also prefer string interpolation to +, s"Unknown UDF input type $c"

s"Unsupported java type $c" seems to be better in this error message because this method is not only designed for UDF.

marmbrus · 2015-04-11T21:48:38Z

Thanks for researching this. Can you address the final comments about avoiding the creation of a new type?

SparkQA · 2015-04-14T16:53:27Z

Test build #30253 has started for PR 5395 at commit 3a8d952.

maropu · 2015-04-14T17:01:10Z

Sorry for the delay. Fixed and plz re-check them.

SparkQA · 2015-04-14T17:09:18Z

Test build #30253 has finished for PR 5395 at commit 3a8d952.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch adds the following new dependencies:
- RoaringBitmap-0.4.5.jar
- activation-1.1.jar
- akka-actor_2.10-2.3.4-spark.jar
- akka-remote_2.10-2.3.4-spark.jar
- akka-slf4j_2.10-2.3.4-spark.jar
- aopalliance-1.0.jar
- arpack_combined_all-0.1.jar
- avro-1.7.7.jar
- breeze-macros_2.10-0.11.2.jar
- breeze_2.10-0.11.2.jar
- chill-java-0.5.0.jar
- chill_2.10-0.5.0.jar
- commons-beanutils-1.7.0.jar
- commons-beanutils-core-1.8.0.jar
- commons-cli-1.2.jar
- commons-codec-1.10.jar
- commons-collections-3.2.1.jar
- commons-compress-1.4.1.jar
- commons-configuration-1.6.jar
- commons-digester-1.8.jar
- commons-httpclient-3.1.jar
- commons-io-2.1.jar
- commons-lang-2.5.jar
- commons-lang3-3.3.2.jar
- commons-math-2.1.jar
- commons-math3-3.4.1.jar
- commons-net-2.2.jar
- compress-lzf-1.0.0.jar
- config-1.2.1.jar
- core-1.1.2.jar
- curator-client-2.4.0.jar
- curator-framework-2.4.0.jar
- curator-recipes-2.4.0.jar
- gmbal-api-only-3.0.0-b023.jar
- grizzly-framework-2.1.2.jar
- grizzly-http-2.1.2.jar
- grizzly-http-server-2.1.2.jar
- grizzly-http-servlet-2.1.2.jar
- grizzly-rcm-2.1.2.jar
- groovy-all-2.3.7.jar
- guava-14.0.1.jar
- guice-3.0.jar
- hadoop-annotations-2.2.0.jar
- hadoop-auth-2.2.0.jar
- hadoop-client-2.2.0.jar
- hadoop-common-2.2.0.jar
- hadoop-hdfs-2.2.0.jar
- hadoop-mapreduce-client-app-2.2.0.jar
- hadoop-mapreduce-client-common-2.2.0.jar
- hadoop-mapreduce-client-core-2.2.0.jar
- hadoop-mapreduce-client-jobclient-2.2.0.jar
- hadoop-mapreduce-client-shuffle-2.2.0.jar
- hadoop-yarn-api-2.2.0.jar
- hadoop-yarn-client-2.2.0.jar
- hadoop-yarn-common-2.2.0.jar
- hadoop-yarn-server-common-2.2.0.jar
- ivy-2.4.0.jar
- jackson-annotations-2.4.0.jar
- jackson-core-2.4.4.jar
- jackson-core-asl-1.8.8.jar
- jackson-databind-2.4.4.jar
- jackson-jaxrs-1.8.8.jar
- jackson-mapper-asl-1.8.8.jar
- jackson-module-scala_2.10-2.4.4.jar
- jackson-xc-1.8.8.jar
- jansi-1.4.jar
- javax.inject-1.jar
- javax.servlet-3.0.0.v201112011016.jar
- javax.servlet-3.1.jar
- javax.servlet-api-3.0.1.jar
- jaxb-api-2.2.2.jar
- jaxb-impl-2.2.3-1.jar
- jcl-over-slf4j-1.7.10.jar
- jersey-client-1.9.jar
- jersey-core-1.9.jar
- jersey-grizzly2-1.9.jar
- jersey-guice-1.9.jar
- jersey-json-1.9.jar
- jersey-server-1.9.jar
- jersey-test-framework-core-1.9.jar
- jersey-test-framework-grizzly2-1.9.jar
- jets3t-0.7.1.jar
- jettison-1.1.jar
- jetty-util-6.1.26.jar
- jline-0.9.94.jar
- jline-2.10.4.jar
- jodd-core-3.6.3.jar
- json4s-ast_2.10-3.2.10.jar
- json4s-core_2.10-3.2.10.jar
- json4s-jackson_2.10-3.2.10.jar
- jsr305-1.3.9.jar
- jtransforms-2.4.0.jar
- jul-to-slf4j-1.7.10.jar
- kryo-2.21.jar
- log4j-1.2.17.jar
- lz4-1.2.0.jar
- management-api-3.0.0-b012.jar
- mesos-0.21.0-shaded-protobuf.jar
- metrics-core-3.1.0.jar
- metrics-graphite-3.1.0.jar
- metrics-json-3.1.0.jar
- metrics-jvm-3.1.0.jar
- minlog-1.2.jar
- netty-3.8.0.Final.jar
- netty-all-4.0.23.Final.jar
- objenesis-1.2.jar
- opencsv-2.3.jar
- oro-2.0.8.jar
- paranamer-2.6.jar
- parquet-column-1.6.0rc3.jar
- parquet-common-1.6.0rc3.jar
- parquet-encoding-1.6.0rc3.jar
- parquet-format-2.2.0-rc1.jar
- parquet-generator-1.6.0rc3.jar
- parquet-hadoop-1.6.0rc3.jar
- parquet-jackson-1.6.0rc3.jar
- protobuf-java-2.4.1.jar
- protobuf-java-2.5.0-spark.jar
- py4j-0.8.2.1.jar
- pyrolite-2.0.1.jar
- quasiquotes_2.10-2.0.1.jar
- reflectasm-1.07-shaded.jar
- scala-compiler-2.10.4.jar
- scala-library-2.10.4.jar
- scala-reflect-2.10.4.jar
- scalap-2.10.4.jar
- scalatest_2.10-2.2.1.jar
- slf4j-api-1.7.10.jar
- slf4j-log4j12-1.7.10.jar
- snappy-java-1.1.1.6.jar
- spark-bagel_2.10-1.4.0-SNAPSHOT.jar
- spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
- spark-core_2.10-1.4.0-SNAPSHOT.jar
- spark-graphx_2.10-1.4.0-SNAPSHOT.jar
- spark-launcher_2.10-1.4.0-SNAPSHOT.jar
- spark-mllib_2.10-1.4.0-SNAPSHOT.jar
- spark-network-common_2.10-1.4.0-SNAPSHOT.jar
- spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
- spark-repl_2.10-1.4.0-SNAPSHOT.jar
- spark-sql_2.10-1.4.0-SNAPSHOT.jar
- spark-streaming_2.10-1.4.0-SNAPSHOT.jar
- spire-macros_2.10-0.7.4.jar
- spire_2.10-0.7.4.jar
- stax-api-1.0.1.jar
- stream-2.7.0.jar
- tachyon-0.5.0.jar
- tachyon-client-0.5.0.jar
- uncommons-maths-1.2.2a.jar
- unused-1.0.0.jar
- xmlenc-0.52.jar
- xz-1.0.jar
- zookeeper-3.4.5.jar

AmplabJenkins · 2015-04-14T17:09:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30253/
Test FAILed.

SparkQA · 2015-04-14T18:08:33Z

Test build #30265 has started for PR 5395 at commit 8e333c7.

SparkQA · 2015-04-14T19:35:22Z

Test build #30265 has finished for PR 5395 at commit 8e333c7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-14T19:35:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30265/
Test PASSed.

marmbrus · 2015-04-16T23:43:23Z

This is still creating a new type. Can we use NullType instead?

maropu · 2015-04-17T00:56:09Z

Missed and fixed. This fix satisfies your point?

SparkQA · 2015-04-17T00:58:30Z

Test build #30445 has started for PR 5395 at commit ee56a0a.

marmbrus · 2015-04-17T01:06:09Z

Yes, LGTM

SparkQA · 2015-04-17T01:53:43Z

Test build #30445 has finished for PR 5395 at commit ee56a0a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-17T01:53:47Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30445/
Test FAILed.

maropu · 2015-04-27T01:51:10Z

cc @marmbrus Could you merge into master? I'll make a PR of SPARK-6912, but it depends on this.

AmplabJenkins · 2015-04-27T18:19:15Z

Can one of the admins verify this patch?

maropu · 2015-05-07T02:03:53Z

cc @marmbrus just a reminder

marmbrus · 2015-05-07T19:50:00Z

The last patch failed tests, no?

marmbrus · 2015-05-07T19:50:03Z

ok to test

AmplabJenkins · 2015-05-07T19:52:13Z

Merged build triggered.

AmplabJenkins · 2015-05-07T19:52:23Z

Merged build started.

SparkQA · 2015-05-07T19:53:06Z

Test build #32142 has started for PR 5395 at commit ee56a0a.

SparkQA · 2015-05-07T21:02:26Z

Test build #32142 has finished for PR 5395 at commit ee56a0a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-07T21:02:30Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-07T21:02:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32142/
Test FAILed.

maropu · 2015-05-12T07:07:38Z

Oh, sorry. I'll fix it.

maropu · 2015-05-15T06:18:31Z

@marmbrus Made a mistake to close this pr, so may I make a new pr because I can't re-open it.

PRs Merged 1. [Internal] Add AppleAwsClientFactory for Mascot (apache#577) 2. Hive: Log new metadata location in commit (apache#4681) 3. change timeout to 120 for now (apache#661) 4. Internal: Add hive_catalog parameter to SparkCatalog (apache#670) 5. Internal: Pull catalog setting to CachedClientPool (apache#673) 6. Core: Defer reading Avro metadata until ManifestFile is read (apache#5206) 7. API: Fix ID assignment in schema merging (apache#5395) 8. AWS: S3OutputStream - failure to close should persist on subsequent close calls (apache#5311) 9. API: Allow schema updates to find fields with case-insensitivity (apache#5440) 10. Spark 3.3: Spark mergeSchema to respect Spark Case Sensitivity Configuration (apache#5441)

chenghao-intel reviewed Apr 7, 2015
View reviewed changes

marmbrus reviewed Apr 11, 2015
View reviewed changes

maropu force-pushed the FixBugInHiveInspectors branch from 02b3a91 to 3a8d952 Compare April 14, 2015 16:49

maropu added 5 commits April 17, 2015 09:51

Support List as a return type in Hive UDF

c31af2e

Add a blank line at the end of UDFToListString

d0a86ff

Apply review comments

5bcb095

Fix code-style errors

a5a2ab9

Remove a new type

ee56a0a

maropu force-pushed the FixBugInHiveInspectors branch from 8e333c7 to ee56a0a Compare April 17, 2015 00:52

maropu closed this May 15, 2015

maropu deleted the FixBugInHiveInspectors branch May 15, 2015 06:07

maropu mentioned this pull request May 15, 2015

[SPARK-6747] [SQL] Support List<> as a return type in Hive UDF #6179

Closed

maropu mentioned this pull request Jul 7, 2015

[SPARK-6747] [SQL] Throw an AnalysisException when unsupported Java list types used in Hive UDF #7248

Closed

[SPARK-6747][SQL] Support List<> as a return type in Hive UDF #5395

[SPARK-6747][SQL] Support List<> as a return type in Hive UDF #5395

Conversation

maropu commented Apr 7, 2015

AmplabJenkins commented Apr 7, 2015

chenghao-intel Apr 7, 2015

Choose a reason for hiding this comment

maropu Apr 8, 2015

Choose a reason for hiding this comment

chenghao-intel commented Apr 7, 2015

marmbrus commented Apr 7, 2015

SparkQA commented Apr 7, 2015

SparkQA commented Apr 7, 2015

AmplabJenkins commented Apr 7, 2015

SparkQA commented Apr 8, 2015

maropu commented Apr 8, 2015

SparkQA commented Apr 8, 2015

AmplabJenkins commented Apr 8, 2015

maropu commented Apr 11, 2015

marmbrus Apr 11, 2015

Choose a reason for hiding this comment

marmbrus Apr 11, 2015

Choose a reason for hiding this comment

maropu Apr 14, 2015

Choose a reason for hiding this comment

marmbrus commented Apr 11, 2015

SparkQA commented Apr 14, 2015

maropu commented Apr 14, 2015

SparkQA commented Apr 14, 2015

AmplabJenkins commented Apr 14, 2015

SparkQA commented Apr 14, 2015

SparkQA commented Apr 14, 2015

AmplabJenkins commented Apr 14, 2015

marmbrus commented Apr 16, 2015

maropu commented Apr 17, 2015

SparkQA commented Apr 17, 2015

marmbrus commented Apr 17, 2015

SparkQA commented Apr 17, 2015

AmplabJenkins commented Apr 17, 2015

maropu commented Apr 27, 2015

AmplabJenkins commented Apr 27, 2015

maropu commented May 7, 2015

marmbrus commented May 7, 2015

marmbrus commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

maropu commented May 12, 2015

maropu commented May 15, 2015