[CARBONDATA-2609] Change RPC implementation to Hadoop RPC framework #2372

jackylk · 2018-06-13T16:15:54Z

In Carbon Search Mode, change the RPC implementation to use Hadoop RPC framework, decoupling from Spark.

Any interfaces changed?
No
Any backward compatibility impacted?
No
Document update required?
No
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA

CarbonDataQA · 2018-06-13T16:21:14Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6331/

CarbonDataQA · 2018-06-13T16:22:55Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5169/

ravipesala · 2018-06-13T16:27:06Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5283/

ravipesala · 2018-06-14T18:29:33Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5293/

CarbonDataQA · 2018-06-14T18:31:23Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6343/

CarbonDataQA · 2018-06-14T18:39:13Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5181/

CarbonDataQA · 2018-06-15T03:45:28Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6347/

ravipesala · 2018-06-15T03:45:43Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5297/

CarbonDataQA · 2018-06-15T03:46:16Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5185/

CarbonDataQA · 2018-06-18T13:17:48Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6359/

CarbonDataQA · 2018-06-18T13:19:41Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5197/

CarbonDataQA · 2018-06-18T13:28:06Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6360/

CarbonDataQA · 2018-06-18T13:30:01Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5198/

ravipesala · 2018-06-18T14:11:35Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5308/

ravipesala · 2018-06-18T15:02:33Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5309/

CarbonDataQA · 2018-06-18T16:23:54Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6362/

CarbonDataQA · 2018-06-18T16:26:01Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5200/

CarbonDataQA · 2018-06-18T16:33:51Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6363/

CarbonDataQA · 2018-06-18T16:35:56Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5201/

ravipesala · 2018-06-18T17:42:28Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5311/

CarbonDataQA · 2018-06-18T17:58:31Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6364/

CarbonDataQA · 2018-06-18T18:05:39Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5202/

ravipesala · 2018-06-18T18:31:36Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5312/

ravipesala · 2018-06-18T19:20:45Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5313/

xubo245 · 2018-06-20T02:38:26Z

hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonRecordReader.java

@@ -80,7 +80,7 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext context)
    }
    // It should use the exists tableBlockInfos if tableBlockInfos of queryModel is not empty
    // otherwise the prune is no use before this method
-    if (!queryModel.isFG()) {
+    if (queryModel.getTableBlockInfos().isEmpty()) {


If use (queryModel.getTableBlockInfos().isEmpty()), when the prune result is empty by FG in search mode, it will use the original TableBlockInfos and execute again, it mean the FG no use in this scenario. So it's better keep it and don't change it to queryModel.getTableBlockInfos().isEmpty()

I have added a new RecordReader called IndexedRecordReader, I am using this one in search mode now. So this problem will not come. I will remove line 83.

CarbonDataQA · 2018-06-20T17:42:56Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5258/

CarbonDataQA · 2018-06-20T18:45:09Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6425/

CarbonDataQA · 2018-06-20T19:43:51Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6428/

ravipesala · 2018-06-21T00:46:18Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5363/

ravipesala · 2018-06-21T00:52:40Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5364/

CarbonDataQA · 2018-06-21T04:20:53Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6433/

CarbonDataQA · 2018-06-21T05:15:01Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5264/

fix test fix test wip wip wip wip wip wip fix test fix test fix test fix test fix test fix comment refactor fix test

CarbonDataQA · 2018-06-22T01:51:02Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6458/

CarbonDataQA · 2018-06-22T02:43:49Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5290/

ravipesala · 2018-06-22T03:45:30Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5383/

xubo245 · 2018-06-22T12:33:58Z

...st/src/test/scala/org/apache/carbondata/spark/testsuite/detailquery/SearchModeTestCase.scala

@@ -137,7 +149,7 @@ class SearchModeTestCase extends QueryTest with BeforeAndAfterAll {
    sql("DROP DATAMAP if exists dm3 ON TABLE main")
  }

-  test("start search mode twice") {
+  ignore("start search mode twice") {


Why ignore it?

This closes #2372

QiangCai · 2018-06-25T01:50:53Z

please close it, already merged into the branch carbonstore
.

This closes #2372

This closes apache#2372

jackylk changed the title ~~[SearchMode] Change RPC implementation to Hadoop RPC framework~~ [CARBONDATA-2609] Change RPC implementation to Hadoop RPC framework Jun 14, 2018

jackylk mentioned this pull request Jun 15, 2018

[CARBONDATA-2613] Support csv based carbon table #2374

Closed

5 tasks

jackylk force-pushed the store-rpc branch from 2780745 to 85189db Compare June 18, 2018 13:15

jackylk mentioned this pull request Jun 19, 2018

[CARBONDATA-2532][Integration] Carbon to support spark 2.3 version #2366

Closed

xubo245 reviewed Jun 20, 2018

View reviewed changes

change RPC implementation to use Hadoop RPC

22f908d

fix test fix test wip wip wip wip wip wip fix test fix test fix test fix test fix test fix comment refactor fix test

jackylk changed the base branch from master to carbonstore June 22, 2018 01:37

jackylk force-pushed the store-rpc branch from 90cc09b to 22f908d Compare June 22, 2018 01:38

fix test

f3dbdf7

xubo245 reviewed Jun 22, 2018

View reviewed changes

asfgit pushed a commit that referenced this pull request Jun 25, 2018

[CARBONDATA-2609] Change RPC implementation to Hadoop RPC framework

fa11138

This closes #2372

jackylk closed this Jun 26, 2018

asfgit pushed a commit that referenced this pull request Jul 18, 2018

[CARBONDATA-2609] Change RPC implementation to Hadoop RPC framework

d9b40bf

This closes #2372

xubo245 pushed a commit to xubo245/carbondata that referenced this pull request Aug 30, 2018

[CARBONDATA-2609] Change RPC implementation to Hadoop RPC framework

e80126a

This closes apache#2372

xubo245 pushed a commit to xubo245/carbondata that referenced this pull request Aug 30, 2018

[CARBONDATA-2609] Change RPC implementation to Hadoop RPC framework

ea04590

This closes apache#2372

xubo245 pushed a commit to xubo245/carbondata that referenced this pull request Sep 3, 2018

[CARBONDATA-2609] Change RPC implementation to Hadoop RPC framework

263ac6a

This closes apache#2372

[CARBONDATA-2609] Change RPC implementation to Hadoop RPC framework #2372

[CARBONDATA-2609] Change RPC implementation to Hadoop RPC framework #2372

Conversation

jackylk commented Jun 13, 2018

CarbonDataQA commented Jun 13, 2018

CarbonDataQA commented Jun 13, 2018

ravipesala commented Jun 13, 2018

ravipesala commented Jun 14, 2018

CarbonDataQA commented Jun 14, 2018

CarbonDataQA commented Jun 14, 2018

CarbonDataQA commented Jun 15, 2018

ravipesala commented Jun 15, 2018

CarbonDataQA commented Jun 15, 2018

CarbonDataQA commented Jun 18, 2018

CarbonDataQA commented Jun 18, 2018

CarbonDataQA commented Jun 18, 2018

CarbonDataQA commented Jun 18, 2018

ravipesala commented Jun 18, 2018

ravipesala commented Jun 18, 2018

CarbonDataQA commented Jun 18, 2018

CarbonDataQA commented Jun 18, 2018

CarbonDataQA commented Jun 18, 2018

CarbonDataQA commented Jun 18, 2018

ravipesala commented Jun 18, 2018

CarbonDataQA commented Jun 18, 2018

CarbonDataQA commented Jun 18, 2018

ravipesala commented Jun 18, 2018

ravipesala commented Jun 18, 2018

xubo245 Jun 20, 2018 • edited

Choose a reason for hiding this comment

jackylk Jun 20, 2018

Choose a reason for hiding this comment

CarbonDataQA commented Jun 20, 2018

CarbonDataQA commented Jun 20, 2018

CarbonDataQA commented Jun 20, 2018

ravipesala commented Jun 21, 2018

ravipesala commented Jun 21, 2018

CarbonDataQA commented Jun 21, 2018

CarbonDataQA commented Jun 21, 2018

CarbonDataQA commented Jun 22, 2018

CarbonDataQA commented Jun 22, 2018

ravipesala commented Jun 22, 2018

xubo245 Jun 22, 2018

Choose a reason for hiding this comment

QiangCai commented Jun 25, 2018

xubo245 Jun 20, 2018 •

edited