[SPARK-9403][SQL] Add codegen support in In and InSet #7893

viirya · 2015-08-03T07:20:00Z

This continues @tarekauel's work in #7778.

… the threshold.

… types.

viirya · 2015-08-03T07:21:05Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/PredicateSuite.scala

+        val value = dataGen.apply()
+        value match {
+          case d: Double if d.isNaN => 0.0d
+          case f: Float if f.isNaN => 0.0f


Double.NaN == Double.NaN is false. So here we don't use it.

SparkQA · 2015-08-03T07:42:50Z

Test build #39530 has finished for PR 7893 at commit 224f18e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class In(value: Expression, list: Seq[Expression]) extends Predicate
- case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with Predicate

SparkQA · 2015-08-03T13:38:47Z

Test build #39543 has finished for PR 7893 at commit 4610eff.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class In(value: Expression, list: Seq[Expression]) extends Predicate
- case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with Predicate

SparkQA · 2015-08-03T17:04:27Z

Test build #39552 has finished for PR 7893 at commit 446bbcd.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class In(value: Expression, list: Seq[Expression]) extends Predicate
- case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with Predicate

rxin · 2015-08-03T19:55:09Z

cc @davies for review

davies · 2015-08-04T05:32:57Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala

@@ -97,7 +101,7 @@ case class Not(child: Expression)
 /**
 * Evaluates to `true` if `list` contains `value`.
 */
-case class In(value: Expression, list: Seq[Expression]) extends Predicate with CodegenFallback {
+case class In(value: Expression, list: Seq[Expression]) extends Predicate {


Existing: Should we check that value and element of list have the same data type?

Is it ok to add a check in CheckAnalysis for this?

It's better to use ExpectedInputType, see https://github.com/apache/spark/pull/7949/files#diff-9853dcf5ce3d2ac1e94d473197ff5768R136

SparkQA · 2015-08-04T12:04:39Z

Test build #39701 has finished for PR 7893 at commit cf4bf41.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class In(value: Expression, list: Seq[Expression]) extends Predicate
- case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with Predicate

viirya · 2015-08-04T13:16:36Z

retest this please.

davies · 2015-08-04T16:55:12Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala

+            ${ev.primitive} = true;
+          }
+        }
+       """).foldLeft("")((a, b) => a + "\n" + b)


We usually use mkString("\n")

davies · 2015-08-04T16:59:03Z

LGTM, except some minor comments.

SparkQA · 2015-08-04T17:32:08Z

Test build #39713 has finished for PR 7893 at commit cf4bf41.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class In(value: Expression, list: Seq[Expression]) extends Predicate
- case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with Predicate

SparkQA · 2015-08-04T19:04:42Z

Test build #208 has finished for PR 7893 at commit cf4bf41.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2015-08-05T06:43:12Z

retest this please.

SparkQA · 2015-08-05T08:03:47Z

Test build #225 has finished for PR 7893 at commit 81ff97b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2015-08-05T08:08:46Z

Jenkins looks like unstable...

viirya · 2015-08-05T08:08:55Z

retest this please.

SparkQA · 2015-08-05T08:22:22Z

Test build #39846 has finished for PR 7893 at commit 81ff97b.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class In(value: Expression, list: Seq[Expression]) extends Predicate
- case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with Predicate

SparkQA · 2015-08-05T08:30:48Z

Test build #1350 has finished for PR 7893 at commit 81ff97b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-05T08:31:37Z

Test build #228 has finished for PR 7893 at commit 81ff97b.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class In(value: Expression, list: Seq[Expression]) extends Predicate
- case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with Predicate

SparkQA · 2015-08-05T08:53:07Z

Test build #39833 has finished for PR 7893 at commit 81ff97b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class In(value: Expression, list: Seq[Expression]) extends Predicate
- case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with Predicate

davies · 2015-08-05T18:38:41Z

LGTM, merging this into master and 1.5 branch, thanks!

This continues tarekauel's work in #7778. Author: Liang-Chi Hsieh <viirya@appier.com> Author: Tarek Auel <tarek.auel@googlemail.com> Closes #7893 from viirya/codegen_in and squashes the following commits: 81ff97b [Liang-Chi Hsieh] For comments. 47761c6 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into codegen_in cf4bf41 [Liang-Chi Hsieh] For comments. f532b3c [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into codegen_in 446bbcd [Liang-Chi Hsieh] Fix bug. b3d0ab4 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into codegen_in 4610eff [Liang-Chi Hsieh] Relax the types of references and update optimizer test. 224f18e [Liang-Chi Hsieh] Beef up the test cases for In and InSet to include all primitive data types. 86dc8aa [Liang-Chi Hsieh] Only convert In to InSet when the number of items in set is more than the threshold. b7ded7e [Tarek Auel] [SPARK-9403][SQL] codeGen in / inSet (cherry picked from commit e1e0587) Signed-off-by: Davies Liu <davies.liu@gmail.com>

tarekbecker and others added 3 commits August 3, 2015 11:44

[SPARK-9403][SQL] codeGen in / inSet

b7ded7e

Only convert In to InSet when the number of items in set is more than…

86dc8aa

… the threshold.

Beef up the test cases for In and InSet to include all primitive data…

224f18e

… types.

viirya reviewed Aug 3, 2015
View reviewed changes

Relax the types of references and update optimizer test.

4610eff

viirya added 2 commits August 3, 2015 23:11

Merge remote-tracking branch 'upstream/master' into codegen_in

b3d0ab4

Fix bug.

446bbcd

davies reviewed Aug 4, 2015
View reviewed changes

viirya added 2 commits August 4, 2015 18:15

Merge remote-tracking branch 'upstream/master' into codegen_in

f532b3c

For comments.

cf4bf41

davies reviewed Aug 4, 2015
View reviewed changes

viirya added 2 commits August 5, 2015 11:24

Merge remote-tracking branch 'upstream/master' into codegen_in

47761c6

For comments.

81ff97b

asfgit closed this in e1e0587 Aug 5, 2015

asfgit pushed a commit that referenced this pull request Aug 5, 2015

Closes #7778 since it is done as #7893.

eb5b8f4

viirya deleted the codegen_in branch December 27, 2023 18:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-9403][SQL] Add codegen support in In and InSet #7893

[SPARK-9403][SQL] Add codegen support in In and InSet #7893

viirya commented Aug 3, 2015

viirya Aug 3, 2015

SparkQA commented Aug 3, 2015

SparkQA commented Aug 3, 2015

SparkQA commented Aug 3, 2015

rxin commented Aug 3, 2015

davies Aug 4, 2015

viirya Aug 4, 2015

davies Aug 4, 2015

davies Aug 4, 2015

SparkQA commented Aug 4, 2015

viirya commented Aug 4, 2015

davies Aug 4, 2015

davies commented Aug 4, 2015

SparkQA commented Aug 4, 2015

SparkQA commented Aug 4, 2015

viirya commented Aug 5, 2015

SparkQA commented Aug 5, 2015

viirya commented Aug 5, 2015

viirya commented Aug 5, 2015

SparkQA commented Aug 5, 2015

SparkQA commented Aug 5, 2015

SparkQA commented Aug 5, 2015

SparkQA commented Aug 5, 2015

davies commented Aug 5, 2015

[SPARK-9403][SQL] Add codegen support in In and InSet #7893

[SPARK-9403][SQL] Add codegen support in In and InSet #7893

Conversation

viirya commented Aug 3, 2015

viirya Aug 3, 2015

Choose a reason for hiding this comment

SparkQA commented Aug 3, 2015

SparkQA commented Aug 3, 2015

SparkQA commented Aug 3, 2015

rxin commented Aug 3, 2015

davies Aug 4, 2015

Choose a reason for hiding this comment

viirya Aug 4, 2015

Choose a reason for hiding this comment

davies Aug 4, 2015

Choose a reason for hiding this comment

davies Aug 4, 2015

Choose a reason for hiding this comment

SparkQA commented Aug 4, 2015

viirya commented Aug 4, 2015

davies Aug 4, 2015

Choose a reason for hiding this comment

davies commented Aug 4, 2015

SparkQA commented Aug 4, 2015

SparkQA commented Aug 4, 2015

viirya commented Aug 5, 2015

SparkQA commented Aug 5, 2015

viirya commented Aug 5, 2015

viirya commented Aug 5, 2015

SparkQA commented Aug 5, 2015

SparkQA commented Aug 5, 2015

SparkQA commented Aug 5, 2015

SparkQA commented Aug 5, 2015

davies commented Aug 5, 2015