[CARBONDATA-1882] select with group by and insertoverwrite to another carbon table #1641

kushalsaha · 2017-12-11T15:47:58Z

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

Any interfaces changed?
No
Any backward compatibility impacted?
No
Document update required?
No
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
Yes
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

CarbonDataQA · 2017-12-11T15:49:04Z

Can one of the admins verify this patch?

CarbonDataQA · 2017-12-11T16:22:30Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1877/

ravipesala · 2017-12-11T17:38:28Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2223/

gvramana · 2017-12-12T05:40:04Z

integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala

@@ -486,6 +486,21 @@ object CarbonDataRDDFactory {
      // if segment is empty then fail the data load


Correct comment

gvramana · 2017-12-12T05:57:30Z

integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala

+                loadDetails.setSegmentStatus(SegmentStatus.MARKED_FOR_DELETE)
+              }
+          }
+          val carbonTablePath = CarbonStorePath


loadTablePreStatusUpdateEvent is not fired,

how about old dictionary to be overwritten?

updatestatus file also needs to be handled accordingly.
Suggest to flow the original flow handling empty segment case

loadTablePreStatusUpdateEvent is not fired, :- it will fire only when the data loading is done . in case of zero record data loading is not done

how about old dictionary to be overwritten :- in case of insert overwrite dictionary is appending

CarbonDataQA · 2017-12-13T12:19:45Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1926/

ravipesala · 2017-12-13T13:29:44Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2256/

ravipesala · 2017-12-13T13:29:54Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2257/

ravipesala · 2017-12-13T14:06:49Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2258/

gvramana · 2017-12-13T15:53:33Z

...t/scala/org/apache/carbondata/spark/testsuite/allqueries/InsertIntoCarbonTableTestCase.scala

+
+  }
+
+  test("insert overwrite in group by scenario with t1 no record and t2 some record") {


Move common code to a function

gvramana · 2017-12-13T15:58:33Z

integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala

@@ -375,7 +375,15 @@ object CarbonDataRDDFactory {
              }
          }
        } else {
-          loadStatus = SegmentStatus.LOAD_FAILURE
+          if (dataFrame.isDefined && updateModel.isEmpty) {


Write comment explaining this

gvramana · 2017-12-13T15:59:28Z

integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala

        LOGGER.audit(s"Data load is failed for " +
                     s"${ carbonLoadModel.getDatabaseName }.${ carbonLoadModel.getTableName }" +
                     " as there is no data to load")
        LOGGER.warn("Cannot write load metadata file as data load failed")
-        throw new Exception("No Data to load")
+


write comment 'as no records loaded in new segment, new segment should be deleted'

CarbonDataQA · 2017-12-13T19:58:28Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1946/

ravipesala · 2017-12-13T23:02:57Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2272/

CarbonDataQA · 2017-12-14T09:29:35Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1959/

ravipesala · 2017-12-14T12:53:44Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2281/

CarbonDataQA · 2017-12-14T14:24:38Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1969/

kushalsaha · 2017-12-14T16:33:31Z

...t/scala/org/apache/carbondata/spark/testsuite/allqueries/InsertIntoCarbonTableTestCase.scala

+
+  test("insert overwrite in group by scenario with t1 no record and t2 some record") {
+    queryExecution("overwriteTable1_noRecord.csv","overwriteTable2_someRecord.csv")
+    sql ("insert overwrite table OverwriteTable_t2 select id,name,sum(salary) as TotalSalary,'98' as age from OverwriteTable_t1 group by id,name,salary")


only insert overwrite query is kept in all the testcases as we handled two different scenario 1) IN GROUP BY CASE
2) WITHOUT GROUP BY CASE if we refactor so one extra method need to be written where again same code will exists

ravipesala · 2017-12-14T18:10:02Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2289/

CarbonDataQA · 2017-12-14T22:01:30Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1980/

ravipesala · 2017-12-14T23:12:14Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2299/

gvramana · 2017-12-15T01:31:09Z

3 load test cases are failing.. plz fix

jackylk · 2017-12-15T09:29:05Z

integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala

@@ -505,7 +516,11 @@ object CarbonDataRDDFactory {
        carbonLoadModel)
      operationContext.setProperty("isOverwrite", overwriteTable)
      OperationListenerBus.getInstance().fireEvent(loadTablePreStatusUpdateEvent, operationContext)
-      val done = updateTableStatus(status, carbonLoadModel, loadStatus, overwriteTable)
+      val done = updateTableStatus(status,


move status to next line

CarbonDataQA · 2017-12-15T09:57:32Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2001/

CarbonDataQA · 2017-12-15T14:17:10Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2010/

ravipesala · 2017-12-15T14:32:06Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2323/

kushalsaha · 2017-12-16T06:29:33Z

Retest SDV please

manishgupta88 · 2017-12-16T06:33:05Z

retest sdv please

ravipesala · 2017-12-16T06:35:39Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2347/

ravipesala · 2017-12-16T06:41:00Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2348/

kushalsaha · 2017-12-16T06:45:22Z

retest sdv please

ravipesala · 2017-12-16T06:46:08Z

retest this please

ravipesala · 2017-12-16T06:55:27Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2349/

ravipesala · 2017-12-16T07:12:24Z

add to whitelist

ravipesala · 2017-12-16T09:00:55Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2353/

CarbonDataQA · 2017-12-16T09:13:50Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/811/

CarbonDataQA · 2017-12-16T09:45:44Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2042/

ravipesala · 2017-12-16T12:53:37Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2359/

CarbonDataQA · 2017-12-16T13:12:40Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/820/

CarbonDataQA · 2017-12-16T13:12:40Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2049/

CarbonDataQA · 2017-12-16T14:45:01Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2051/

ravipesala · 2017-12-16T14:47:42Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2362/

CarbonDataQA · 2017-12-16T15:30:15Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/823/

gvramana · 2017-12-16T16:41:34Z

LGTM

CarbonDataQA · 2017-12-16T16:46:17Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/826/

… carbon table This closes apache#1641

kushalsaha changed the title ~~select with group by and insertoverwrite to another carbon table~~ [CARBONDATA-1882] select with group by and insertoverwrite to another carbon table Dec 11, 2017

gvramana reviewed Dec 12, 2017

View reviewed changes

dhatchayani force-pushed the DTS_overwrite branch 2 times, most recently from 07c3f12 to caff22d Compare December 13, 2017 11:12

dhatchayani force-pushed the DTS_overwrite branch from caff22d to 65aae37 Compare December 13, 2017 15:49

gvramana reviewed Dec 13, 2017

View reviewed changes

dhatchayani force-pushed the DTS_overwrite branch from 65aae37 to d750892 Compare December 14, 2017 09:01

kushalsaha force-pushed the DTS_overwrite branch from d750892 to 87fb160 Compare December 14, 2017 12:05

kushalsaha force-pushed the DTS_overwrite branch from 87fb160 to c42461b Compare December 14, 2017 16:26

kushalsaha commented Dec 14, 2017

View reviewed changes

kushalsaha force-pushed the DTS_overwrite branch from c42461b to ba1dddb Compare December 15, 2017 07:17

jackylk reviewed Dec 15, 2017

View reviewed changes

kushalsaha force-pushed the DTS_overwrite branch from 2dbe03a to ab5e467 Compare December 15, 2017 10:02

kushalsaha force-pushed the DTS_overwrite branch from ab5e467 to b974499 Compare December 16, 2017 09:35

SDV testCase fixes

43664ec

kushalsaha force-pushed the DTS_overwrite branch from b974499 to 43664ec Compare December 16, 2017 13:20

asfgit closed this in 4a7fc66 Dec 16, 2017

jatin9896 pushed a commit to jatin9896/incubator-carbondata that referenced this pull request Jan 5, 2018

[CARBONDATA-1882] select with group by and insertoverwrite to another…

d4870ea

… carbon table This closes apache#1641

anubhav100 pushed a commit to anubhav100/incubator-carbondata that referenced this pull request Jun 22, 2018

[CARBONDATA-1882] select with group by and insertoverwrite to another…

e682afb

… carbon table This closes apache#1641

		@@ -486,6 +486,21 @@ object CarbonDataRDDFactory {
		// if segment is empty then fail the data load


		}

		test("insert overwrite in group by scenario with t1 no record and t2 some record") {

[CARBONDATA-1882] select with group by and insertoverwrite to another carbon table #1641

[CARBONDATA-1882] select with group by and insertoverwrite to another carbon table #1641

Conversation

kushalsaha commented Dec 11, 2017

CarbonDataQA commented Dec 11, 2017

CarbonDataQA commented Dec 11, 2017

ravipesala commented Dec 11, 2017

gvramana Dec 12, 2017

Choose a reason for hiding this comment

gvramana Dec 12, 2017

Choose a reason for hiding this comment

kushalsaha Dec 12, 2017

Choose a reason for hiding this comment

CarbonDataQA commented Dec 13, 2017

ravipesala commented Dec 13, 2017

ravipesala commented Dec 13, 2017

ravipesala commented Dec 13, 2017

gvramana Dec 13, 2017

Choose a reason for hiding this comment

gvramana Dec 13, 2017

Choose a reason for hiding this comment

gvramana Dec 13, 2017

Choose a reason for hiding this comment

CarbonDataQA commented Dec 13, 2017

ravipesala commented Dec 13, 2017

CarbonDataQA commented Dec 14, 2017

ravipesala commented Dec 14, 2017

CarbonDataQA commented Dec 14, 2017

kushalsaha Dec 14, 2017 • edited

Choose a reason for hiding this comment

ravipesala commented Dec 14, 2017

CarbonDataQA commented Dec 14, 2017

ravipesala commented Dec 14, 2017

gvramana commented Dec 15, 2017

jackylk Dec 15, 2017

Choose a reason for hiding this comment

CarbonDataQA commented Dec 15, 2017

CarbonDataQA commented Dec 15, 2017

ravipesala commented Dec 15, 2017

kushalsaha commented Dec 16, 2017

manishgupta88 commented Dec 16, 2017

ravipesala commented Dec 16, 2017

ravipesala commented Dec 16, 2017

kushalsaha commented Dec 16, 2017

ravipesala commented Dec 16, 2017

ravipesala commented Dec 16, 2017

ravipesala commented Dec 16, 2017

ravipesala commented Dec 16, 2017

CarbonDataQA commented Dec 16, 2017

CarbonDataQA commented Dec 16, 2017

ravipesala commented Dec 16, 2017

CarbonDataQA commented Dec 16, 2017

CarbonDataQA commented Dec 16, 2017

CarbonDataQA commented Dec 16, 2017

ravipesala commented Dec 16, 2017

CarbonDataQA commented Dec 16, 2017

gvramana commented Dec 16, 2017

CarbonDataQA commented Dec 16, 2017

kushalsaha Dec 14, 2017 •

edited