[CARBONDATA-530]Modified optimizer to place decoder on top of limit in case of sort and limt.

[ashokblend]

Problem
While optimizing plan of query which is having order by and limit, we put outer decoder below limit. Because of this spark is not able to optimize the plan.

Solution
We need to optimize the plan in such a way that outer decoder comes on top of limit in above case. as given below

|== Physical Plan == |
|CarbonDictionaryDecoder [CarbonDecoderRelation(Map(name#3 -> name#3),CarbonDatasourceRelation(default.dict,None))], ExcludeProfile(ArrayBuffer(name#3)), CarbonAliasDecoderRelation() |
| TakeOrderedAndProject(limit=2, orderBy=[name#3 ASC], output=[name#3]) |
| ConvertToSafe |
| CarbonDictionaryDecoder [CarbonDecoderRelation(Map(name#3 -> name#3),CarbonDatasourceRelation(default.dict,None))], IncludeProfile(ArrayBuffer(name#3)), CarbonAliasDecoderRelation() |
| CarbonScan [name#3], (CarbonRelation default, dict, CarbonMetaData(ArrayBuffer(name),ArrayBuffer(default_dummy_measure),org.apache.carbondata.core.carbon.metadata.schema.table.CarbonTable@62b3dcd6,DictionaryMap(Map(name -> true))), org.apache.carbondata.spark.merger.TableMeta@e67983a, None), [(name#3 = hello)], false|
| |
|Code Generation: true |
+----------------------

[ravipesala]

LGTM, But please add same code in CarbonLateDecodeRule as well.

[CarbonDataQA]

Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/155/

[CarbonDataQA]

Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/165/

[jackylk]

Seems CI is failing for -Pspark-2.0

[ERROR] /home/travis/build/apache/incubator-carbondata/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonLateDecodeRule.scala:142: error: not found: type Limit
[INFO]         case limit: Limit =>

[CarbonDataQA]

Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/253/

[ashokblend]

@jackylk is there any way to get build url for spark-2.0 as we have for spark-1.5

[jackylk]

@ashokblend, please check

- select imei,series from Carbon_automation_test where series='7Series' order by imei limit 10 *** FAILED ***

[CarbonDataQA]

Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/335/

[CarbonDataQA]

Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1079/

[chenliang613]

please change the title as per the format: [CARBONDATA-issue number>] Title of the pull request (need to add a blank)

[ravipesala]

retest this please

[CarbonDataQA]

Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1173/

[ravipesala]

@ashokblend Spark2.1 testcases are failing, can you fix it.

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/80/
Test FAILed.

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-2.1/231/
Test PASSed.

[CarbonDataQA]

Can one of the admins verify this patch?

[asfgit]

Can one of the admins verify this patch?

[asfgit]

Can one of the admins verify this patch?

[CarbonDataQA]

SDV Build Failed with Spark 2.1, Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/65/

[CarbonDataQA]

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/636/

[ravipesala]

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1267/

[ashokblend]

closing it.