[SPARK-3911] [SQL] HiveSimpleUdf can not be optimized in constant folding #2771

chenghao-intel · 2014-10-11T16:16:24Z

explain extended select cos(null) from src limit 1;

outputs:

 Project [HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFCos(null) AS c_0#5]
  MetastoreRelation default, src, None

== Optimized Logical Plan ==
Limit 1
 Project [HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFCos(null) AS c_0#5]
  MetastoreRelation default, src, None

== Physical Plan ==
Limit 1
 Project [HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFCos(null) AS c_0#5]
  HiveTableScan [], (MetastoreRelation default, src, None), None

After patching this PR it outputs

== Parsed Logical Plan ==
Limit 1
 Project ['cos(null) AS c_0#0]
  UnresolvedRelation None, src, None

== Analyzed Logical Plan ==
Limit 1
 Project [HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFCos(null) AS c_0#0]
  MetastoreRelation default, src, None

== Optimized Logical Plan ==
Limit 1
 Project [null AS c_0#0]
  MetastoreRelation default, src, None

== Physical Plan ==
Limit 1
 Project [null AS c_0#0]
  HiveTableScan [], (MetastoreRelation default, src, None), None

AmplabJenkins · 2014-10-11T16:17:06Z

Can one of the admins verify this patch?

SparkQA · 2014-10-11T16:19:38Z

QA tests have started for PR 2771 at commit cc3091e.

This patch merges cleanly.

SparkQA · 2014-10-11T17:04:43Z

QA tests have finished for PR 2771 at commit cc3091e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-11T17:04:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21647/Test PASSed.

liancheng · 2014-10-13T11:46:56Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala

+  @transient
+  protected lazy val isUDFDeterministic = {
+    val udfType = function.getClass().getAnnotation(classOf[HiveUDFType])
+    (udfType != null && udfType.deterministic())


Nit: redundant parenthesis.

liancheng · 2014-10-13T11:53:05Z

This LGTM. Would you mind to add some tests? Probably in ExpressionOptimizationSuite. Thanks.

chenghao-intel · 2014-10-17T03:11:26Z

Thank you @liancheng , I've updated the code.

chenghao-intel · 2014-10-17T07:09:50Z

test this please.

marmbrus · 2014-10-21T00:05:34Z

ok to test

SparkQA · 2014-10-21T00:09:41Z

QA tests have started for PR 2771 at commit 405d9cf.

This patch merges cleanly.

SparkQA · 2014-10-21T00:13:15Z

QA tests have finished for PR 2771 at commit 405d9cf.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-21T00:13:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21940/
Test FAILed.

liancheng · 2014-10-22T10:04:27Z

retest this please

SparkQA · 2014-10-22T10:09:44Z

QA tests have started for PR 2771 at commit 405d9cf.

This patch merges cleanly.

SparkQA · 2014-10-22T11:02:07Z

QA tests have finished for PR 2771 at commit 405d9cf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-22T11:02:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22033/
Test PASSed.

chenghao-intel · 2014-10-23T00:42:15Z

@marmbrus @liancheng any more comments on this?

liancheng · 2014-10-23T16:38:10Z

sql/hive/src/test/scala/org/apache/spark/sql/QueryTest.scala

@@ -74,4 +76,30 @@ class QueryTest extends FunSuite {
      """.stripMargin)
    }
  }
+
+  // The following copy is copied from org.apache.spark.sql.catalyst.plans.PlanTest


How about making QueryTest inherit from PlanTest instead? Just like what we did in another PlanTest in sql/core.

SparkQA · 2014-10-25T08:07:40Z

Test build #445 has started for PR 2771 at commit 405d9cf.

This patch merges cleanly.

SparkQA · 2014-10-25T10:07:40Z

Test build #445 timed out for PR 2771 at commit 405d9cf after a configured wait of 120m.

SparkQA · 2014-10-26T22:04:56Z

Test build #457 has started for PR 2771 at commit 405d9cf.

This patch merges cleanly.

SparkQA · 2014-10-26T22:31:51Z

Test build #461 has started for PR 2771 at commit 405d9cf.

This patch merges cleanly.

SparkQA · 2014-10-26T22:37:00Z

Test build #457 has finished for PR 2771 at commit 405d9cf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-10-27T00:31:51Z

Test build #461 timed out for PR 2771 at commit 405d9cf after a configured wait of 120m.

chenghao-intel · 2014-10-27T06:49:19Z

test this please.

SparkQA · 2014-10-27T06:54:48Z

Test build #22280 has started for PR 2771 at commit 1379c73.

This patch merges cleanly.

SparkQA · 2014-10-27T07:45:27Z

Test build #22280 has finished for PR 2771 at commit 1379c73.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-27T07:45:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22280/
Test PASSed.

chenghao-intel · 2014-10-28T01:35:47Z

@marmbrus @liancheng any more comment on this?

marmbrus · 2014-10-28T03:42:31Z

Thanks! Merged to master.

liancheng reviewed Oct 13, 2014
View reviewed changes

chenghao-intel force-pushed the hive_udf_constant_folding branch from cc3091e to 405d9cf Compare October 17, 2014 03:09

liancheng reviewed Oct 23, 2014
View reviewed changes

chenghao-intel added 3 commits October 27, 2014 14:33

support constant folding for HiveSimpleUdf

01609ff

add unit test for hive simple udf constant folding

1e52dda

duplicate the PlanTest with catalyst/plans/PlanTest

1379c73

chenghao-intel force-pushed the hive_udf_constant_folding branch from 405d9cf to 1379c73 Compare October 27, 2014 06:48

asfgit closed this in 418ad83 Oct 28, 2014

chenghao-intel deleted the hive_udf_constant_folding branch July 2, 2015 08:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-3911] [SQL] HiveSimpleUdf can not be optimized in constant folding #2771

[SPARK-3911] [SQL] HiveSimpleUdf can not be optimized in constant folding #2771

chenghao-intel commented Oct 11, 2014

AmplabJenkins commented Oct 11, 2014

SparkQA commented Oct 11, 2014

SparkQA commented Oct 11, 2014

AmplabJenkins commented Oct 11, 2014

liancheng Oct 13, 2014

liancheng commented Oct 13, 2014

chenghao-intel commented Oct 17, 2014

chenghao-intel commented Oct 17, 2014

marmbrus commented Oct 21, 2014

SparkQA commented Oct 21, 2014

SparkQA commented Oct 21, 2014

AmplabJenkins commented Oct 21, 2014

liancheng commented Oct 22, 2014

SparkQA commented Oct 22, 2014

SparkQA commented Oct 22, 2014

AmplabJenkins commented Oct 22, 2014

chenghao-intel commented Oct 23, 2014

liancheng Oct 23, 2014

SparkQA commented Oct 25, 2014

SparkQA commented Oct 25, 2014

SparkQA commented Oct 26, 2014

SparkQA commented Oct 26, 2014

SparkQA commented Oct 26, 2014

SparkQA commented Oct 27, 2014

chenghao-intel commented Oct 27, 2014

SparkQA commented Oct 27, 2014

SparkQA commented Oct 27, 2014

AmplabJenkins commented Oct 27, 2014

chenghao-intel commented Oct 28, 2014

marmbrus commented Oct 28, 2014

[SPARK-3911] [SQL] HiveSimpleUdf can not be optimized in constant folding #2771

[SPARK-3911] [SQL] HiveSimpleUdf can not be optimized in constant folding #2771

Conversation

chenghao-intel commented Oct 11, 2014

AmplabJenkins commented Oct 11, 2014

SparkQA commented Oct 11, 2014

SparkQA commented Oct 11, 2014

AmplabJenkins commented Oct 11, 2014

liancheng Oct 13, 2014

Choose a reason for hiding this comment

liancheng commented Oct 13, 2014

chenghao-intel commented Oct 17, 2014

chenghao-intel commented Oct 17, 2014

marmbrus commented Oct 21, 2014

SparkQA commented Oct 21, 2014

SparkQA commented Oct 21, 2014

AmplabJenkins commented Oct 21, 2014

liancheng commented Oct 22, 2014

SparkQA commented Oct 22, 2014

SparkQA commented Oct 22, 2014

AmplabJenkins commented Oct 22, 2014

chenghao-intel commented Oct 23, 2014

liancheng Oct 23, 2014

Choose a reason for hiding this comment

SparkQA commented Oct 25, 2014

SparkQA commented Oct 25, 2014

SparkQA commented Oct 26, 2014

SparkQA commented Oct 26, 2014

SparkQA commented Oct 26, 2014

SparkQA commented Oct 27, 2014

chenghao-intel commented Oct 27, 2014

SparkQA commented Oct 27, 2014

SparkQA commented Oct 27, 2014

AmplabJenkins commented Oct 27, 2014

chenghao-intel commented Oct 28, 2014

marmbrus commented Oct 28, 2014