[CARBONDATA-1703] Refactored code for creation of filter for IN expression #1650

geetikagupta16 · 2017-12-13T06:36:36Z

Problem: There was a difference in result set count of carbon and hive tables

Analysis: This difference was due to the creation of filters in hive and carbon. In hive 'NULL' is not added in filter creation for IN expression whereas in carbon we were adding 'NULL' also in filter creation

Solution: Removed addition of 'NULL' values in case of IN filter creation

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

Any interfaces changed? No
Any backward compatibility impacted? No
Document update required? No
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

CarbonDataQA · 2017-12-13T06:50:12Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1914/

CarbonDataQA · 2017-12-13T07:16:38Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/685/

ravipesala · 2017-12-13T08:29:22Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2247/

CarbonDataQA · 2017-12-13T13:24:56Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/703/

CarbonDataQA · 2017-12-13T14:01:32Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1932/

ravipesala · 2017-12-13T17:19:28Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2263/

CarbonDataQA · 2017-12-14T14:33:02Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/741/

CarbonDataQA · 2017-12-14T15:51:25Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1972/

ravipesala · 2017-12-14T19:25:39Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2291/

sounakr · 2017-12-15T07:28:51Z

integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala

-              convertToJavaList(values.map(f => getCarbonLiteralExpression(name, f)).toList))))
+          if (values.length == 1 && values(0) == null) {
+            Some(new InExpression(getCarbonExpression(name),
+              new ListExpression(


Please make the In and Not In filter as par of TransformExpression. For e.g. Please look into line no 321-323..when it was corrected there it got missed in createFilter.

I have made the null check for IN expression in line no. 89

why two conditions length=1 and others when logic is same. only else condition should be sufficient

Not In case when null comes, in method TransformExpression (as sounak pointed), the behaviour was similar to falseExpression. So we have to conform not IN null case with hive to decide the behaviour.

Please add test cases for In and Not in case with null expression and compare behaviour with hive.

can isNullLiteral(_) possible anytime? as it is in TransformExpression method.

The 1 condition was added when we have only NULL in our IN expression for that case, the Left expression was becoming NULL. So to avoid that we added this case.

CarbonDataQA · 2017-12-15T17:16:46Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/788/

CarbonDataQA · 2017-12-15T20:29:31Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2022/

ravipesala · 2017-12-15T20:32:04Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2334/

sraghunandan · 2017-12-16T07:02:55Z

ok to test

CarbonDataQA · 2017-12-16T08:18:04Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/809/

ravipesala · 2017-12-16T08:19:03Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2351/

CarbonDataQA · 2017-12-16T08:48:25Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2040/

2. Added related test cases

CarbonDataQA · 2017-12-18T08:18:07Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2086/

CarbonDataQA · 2017-12-18T08:22:03Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/861/

ravipesala · 2017-12-18T09:13:15Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2384/

geetikagupta16 · 2017-12-18T12:28:15Z

retest this please

ravipesala · 2017-12-18T14:30:33Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2392/

CarbonDataQA · 2017-12-18T14:37:47Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/875/

CarbonDataQA · 2017-12-18T14:51:21Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2100/

CarbonDataQA · 2017-12-18T15:53:24Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/878/

CarbonDataQA · 2017-12-18T16:13:38Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2103/

CarbonDataQA · 2017-12-20T12:16:28Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/945/

CarbonDataQA · 2017-12-20T12:17:02Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2174/

geetikagupta16 · 2017-12-22T07:54:53Z

@gvramana Please review again, as I have made the required changes

sounakr · 2017-12-27T15:12:32Z

integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala

-            new ListExpression(
-              convertToJavaList(values.map(f => getCarbonLiteralExpression(name, f)).toList))))
+          if (values.length == 1 && values(0) == null) {
+            Some(new InExpression(getCarbonExpression(name),


In case there are only one value in In list and that is also qualifies as Null, then why cant we return False expression just like Not In case. i.e. Some(new FalseExpression(getCarbonExpression(name))).

sounakr · 2017-12-27T15:13:39Z

...est/scala/org/apache/carbondata/spark/testsuite/detailquery/ExpressionWithNullTestCase.scala

@@ -75,6 +78,9 @@ class ExpressionWithNullTestCase extends QueryTest with BeforeAndAfterAll {
    checkAnswer(sql("select * from expression_test where id not in ('2')"), sql("select * from expression_test_hive where id not in ('2')"))
    checkAnswer(sql("select * from expression_test where id not in (cast('2' as int))"), sql("select * from expression_test_hive where id not in (cast('2' as int))"))
 //    checkAnswer(sql("select * from expression_test where id not in (cast('null' as int))"), sql("select * from expression_test_hive where id not in (cast('null' as int))"))


Please uncomment the test cases and rerun it once.

sounakr · 2017-12-27T15:18:11Z

...est/scala/org/apache/carbondata/spark/testsuite/detailquery/ExpressionWithNullTestCase.scala

@@ -75,6 +78,9 @@ class ExpressionWithNullTestCase extends QueryTest with BeforeAndAfterAll {
    checkAnswer(sql("select * from expression_test where id not in ('2')"), sql("select * from expression_test_hive where id not in ('2')"))
    checkAnswer(sql("select * from expression_test where id not in (cast('2' as int))"), sql("select * from expression_test_hive where id not in (cast('2' as int))"))
 //    checkAnswer(sql("select * from expression_test where id not in (cast('null' as int))"), sql("select * from expression_test_hive where id not in (cast('null' as int))"))
+    checkAnswer(sql("select * from expression_test where id not in (1,2,NULL)"), sql("select * from expression_test_hive where id not in (1,2,NULL)"))
+    checkAnswer(sql("select * from expression_test where id not in (NULL)"), sql("select * from expression_test_hive where id not in (NULL)"))


Please add some more test cases with explicit cast and with Nulls

sounakr · 2017-12-27T15:26:17Z

integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala

+          if (values.contains(null)) {
+            Some(new FalseExpression(getCarbonExpression(name)))
+          } else {
+            Some(new NotInExpression(getCarbonExpression(name),


I think similar changes are required in transformExpression also. In case Cast is not resolved by Spark then transformExpression will handle those expression. But prior doing any changes we need supporting test cases.

Can you please provide scenario for the same

…dded test cases

CarbonDataQA · 2017-12-28T06:58:44Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2387/

CarbonDataQA · 2017-12-28T07:01:41Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1171/

CarbonDataQA · 2017-12-28T07:02:39Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2394/

geetikagupta16 · 2017-12-28T11:39:28Z

retest this please

CarbonDataQA · 2017-12-28T13:03:40Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2409/

CarbonDataQA · 2017-12-28T14:02:32Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1185/

ravipesala · 2017-12-28T23:27:28Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2593/

ravipesala · 2017-12-29T00:57:59Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2594/

geetikagupta16 · 2017-12-29T05:21:48Z

retest this please

jackylk · 2017-12-29T08:35:28Z

@geetikagupta16 can you add more description of this bug? what is the bug and what is the root analysis of it?

geetikagupta16 · 2017-12-29T09:16:12Z

@jackylk I have updated the PR description please check

CarbonDataQA · 2017-12-29T11:24:56Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2436/

CarbonDataQA · 2017-12-29T13:34:41Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1212/

sounakr · 2018-01-03T10:01:01Z

LGTM

sraghunandan · 2018-01-04T04:10:59Z

retest sdv please

ravipesala · 2018-01-04T09:18:42Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2696/

gvramana · 2018-01-09T10:21:30Z

LGTM

This closes apache#1650

Refactored code for creation of filter for IN expression

81743ac

Refactor code to handle null values in IN expression

34f181d

Removed scalastyle warning

13bcc59

sounakr approved these changes Dec 15, 2017

View reviewed changes

REfactored code for NOT IN filter

8185286

1. Refactored code for handling null values in NOT IN filter

998c101

2. Added related test cases

Resolved test cases for null values in NOT In filter

95e55e8

sounakr reviewed Dec 27, 2017

View reviewed changes

Refactored code for creation of in filter with only null values and a…

0c9dabe

…dded test cases

geetikagupta16 force-pushed the CARBONDATA-1703 branch from 3884132 to 0c9dabe Compare December 28, 2017 06:48

asfgit closed this in 082a355 Jan 9, 2018

anubhav100 pushed a commit to anubhav100/incubator-carbondata that referenced this pull request Jun 22, 2018

[CARBONDATA-1703] Fix null handling in "IN" and "NOT IN" expressions

ea84c9c

This closes apache#1650

[CARBONDATA-1703] Refactored code for creation of filter for IN expression #1650

[CARBONDATA-1703] Refactored code for creation of filter for IN expression #1650

Conversation

geetikagupta16 commented Dec 13, 2017 • edited

CarbonDataQA commented Dec 13, 2017

CarbonDataQA commented Dec 13, 2017

ravipesala commented Dec 13, 2017

CarbonDataQA commented Dec 13, 2017

CarbonDataQA commented Dec 13, 2017

ravipesala commented Dec 13, 2017

CarbonDataQA commented Dec 14, 2017

CarbonDataQA commented Dec 14, 2017

ravipesala commented Dec 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gvramana Dec 16, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA commented Dec 15, 2017

CarbonDataQA commented Dec 15, 2017

ravipesala commented Dec 15, 2017

sraghunandan commented Dec 16, 2017

CarbonDataQA commented Dec 16, 2017

ravipesala commented Dec 16, 2017

CarbonDataQA commented Dec 16, 2017

CarbonDataQA commented Dec 18, 2017

CarbonDataQA commented Dec 18, 2017

ravipesala commented Dec 18, 2017

geetikagupta16 commented Dec 18, 2017

ravipesala commented Dec 18, 2017

CarbonDataQA commented Dec 18, 2017

CarbonDataQA commented Dec 18, 2017

CarbonDataQA commented Dec 18, 2017

CarbonDataQA commented Dec 18, 2017

CarbonDataQA commented Dec 20, 2017

CarbonDataQA commented Dec 20, 2017

geetikagupta16 commented Dec 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA commented Dec 28, 2017

CarbonDataQA commented Dec 28, 2017

CarbonDataQA commented Dec 28, 2017

geetikagupta16 commented Dec 28, 2017

CarbonDataQA commented Dec 28, 2017

CarbonDataQA commented Dec 28, 2017

ravipesala commented Dec 28, 2017

ravipesala commented Dec 29, 2017

geetikagupta16 commented Dec 29, 2017

jackylk commented Dec 29, 2017

geetikagupta16 commented Dec 29, 2017

CarbonDataQA commented Dec 29, 2017

CarbonDataQA commented Dec 29, 2017

sounakr commented Jan 3, 2018

sraghunandan commented Jan 4, 2018

ravipesala commented Jan 4, 2018

gvramana commented Jan 9, 2018

geetikagupta16 commented Dec 13, 2017 •

edited

gvramana Dec 16, 2017 •

edited