[CARBONDATA-1954] [Pre-Aggregate] CarbonHiveMetastore updated while dropping the Pre-Aggregate table & code refactored #1743

rahulk2 · 2017-12-29T14:41:39Z

This PR contains :
1. To update CarbonHiveMetastore similar function was already there . Removed duplicate function defination and updated the caller.
2. code refactored so that during droping a pre-aggregate table only metadata will be deleted if processMetadata() is called.

Any interfaces changed? No
Any backward compatibility impacted? No
Document update required? No
Testing done - Yes, test case added
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA

ravipesala · 2017-12-29T15:58:00Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2633/

CarbonDataQA · 2017-12-29T17:38:48Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2447/

CarbonDataQA · 2017-12-29T22:44:26Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1227/

rahulk2 · 2018-01-02T03:35:25Z

retest sdv please

ravipesala · 2018-01-02T04:17:28Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2645/

gvramana · 2018-01-03T10:34:23Z

...n-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/TestDataMapCommand.scala

@@ -76,6 +78,24 @@ class TestDataMapCommand extends QueryTest with BeforeAndAfterAll {
    assert(dataMapSchemaList.get(2).getChildSchema.getTableName.equals("datamaptest_datamap3"))
  }

+  test("check hivemetastore after drop datamap") {
+    CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_HIVE_SCHEMA_META_STORE,


reset this property

gvramana · 2018-01-03T10:35:24Z

...n-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/TestDataMapCommand.scala

+    var dataMapSchemaList = table.getTableInfo.getDataMapSchemaList
+    assert(dataMapSchemaList.size() == 1)
+
+    sql("drop datamap if exists datamap_hiveMetaStoreTable on table hiveMetaStoreTable")


remove if exists

drop main table also

Use get datamap list to find if datamap table is dropped

gvramana · 2018-01-03T11:09:18Z

integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetaStore.scala

@@ -194,6 +194,11 @@ class CarbonHiveMetaStore extends CarbonFileMetastore {
      newTablePath)
    val dbName = oldTableIdentifier.getDatabaseName
    val tableName = oldTableIdentifier.getTableName
+    val schemaParts = CarbonUtil.convertToMultiGsonStrings(wrapperTableInfo, "=", "'", "")


Not required this function use above function , updateHiveMetaStoreForAlter

gvramana · 2018-01-03T11:10:15Z

change PR title and issue title to describe the problem

CarbonDataQA · 2018-01-03T15:26:22Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2513/

CarbonDataQA · 2018-01-03T21:46:24Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1296/

ravipesala · 2018-01-03T22:32:25Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2681/

jackylk · 2018-01-04T09:13:10Z

.../spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/DataMapListeners.scala

@@ -65,7 +65,7 @@ object DropDataMapPostListener extends OperationEventListener {
          Some(dataMapSchema.get.getRelationIdentifier.getDatabaseName),
          dataMapSchema.get.getRelationIdentifier.getTableName,
          dropChildTable = true
-        ).run(sparkSession)
+        ).processMetadata(sparkSession)


I think it is better to move this invokation to CarbonDropDataMapCommand.processMetadata

@jackylk code is moved to CarbonDropDataMapCommand

jackylk · 2018-01-04T09:16:13Z

...n-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/TestDataMapCommand.scala

+          CarbonCommonConstants.ENABLE_HIVE_SCHEMA_META_STORE_DEFAULT)
+    }
+  }
+


Beside CarbonDropDataMapCommand.scala, CarbonDropTableCommand.scala also has problem that processMetadata is incorrect if there are datamaps associated with the fact table. Please modify that in this PR also and add testcase

CarbonDropDataMapCommand also modified and test case added.

CarbonDataQA · 2018-01-04T13:49:20Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2548/

CarbonDataQA · 2018-01-04T18:42:40Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1323/

ravipesala · 2018-01-04T20:48:37Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2718/

ravipesala · 2018-01-05T08:49:10Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2743/

rahulk2 · 2018-01-05T08:54:50Z

retest sdv please

ravipesala · 2018-01-05T10:16:43Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2745/

CarbonDataQA · 2018-01-05T10:42:46Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2580/

CarbonDataQA · 2018-01-05T14:47:27Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2588/

ravipesala · 2018-01-05T15:06:36Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2751/

CarbonDataQA · 2018-01-05T16:21:36Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1353/

ravipesala · 2018-01-07T09:07:53Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2780/

CarbonDataQA · 2018-01-07T10:29:04Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2618/

CarbonDataQA · 2018-01-07T10:29:04Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1383/

ravipesala · 2018-01-07T11:08:15Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2781/

CarbonDataQA · 2018-01-07T12:29:04Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1384/

CarbonDataQA · 2018-01-07T12:29:04Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2619/

ravipesala · 2018-01-07T15:11:28Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2782/

CarbonDataQA · 2018-01-07T16:29:04Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2620/

CarbonDataQA · 2018-01-07T16:29:04Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1385/

…aggregate table & code refactored to process metadata only if processMetadata() is called for preaggregate table

CarbonDataQA · 2018-01-08T14:14:52Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1404/

ravipesala · 2018-01-08T15:27:47Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2798/

CarbonDataQA · 2018-01-08T16:18:58Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2639/

ravipesala · 2018-01-08T16:37:59Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2800/

ravipesala · 2018-01-08T16:50:58Z

LGTM

CarbonDataQA · 2018-01-08T18:25:06Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1410/

CarbonDataQA · 2018-01-08T21:33:11Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2646/

xubo245 · 2018-01-25T13:45:42Z

...n-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/TestDataMapCommand.scala

+      CarbonProperties.getInstance()
+        .addProperty(CarbonCommonConstants.ENABLE_HIVE_SCHEMA_META_STORE,
+          "true")
+      sql("drop datamap if exists datamap_hiveMetaStoreTable on table hiveMetaStoreTable")


where is hiveMetaStoreTable?

…ropping the Pre-Aggregate table & code refactored 1. To update CarbonHiveMetastore similar function was already there . Removed duplicate function defination and updated the caller. 2. code refactored so that during droping a pre-aggregate table only metadata will be deleted if processMetadata() is called. This closes apache#1743

gvramana reviewed Jan 3, 2018

View reviewed changes

rahulk2 changed the title ~~[CARBONDATA-1954] HiveMetastore updated while dropping the table & code refactored~~ [CARBONDATA-1954] [Pre-Aggregate] CarbonHiveMetastore updated while dropping the Pre-Aggregate table & code refactored Jan 3, 2018

rahulk2 force-pushed the pre_agg_issue branch from 24ec914 to 30b5fc0 Compare January 3, 2018 12:30

jackylk reviewed Jan 4, 2018

View reviewed changes

rahulk2 force-pushed the pre_agg_issue branch from 30b5fc0 to 31b50ef Compare January 4, 2018 10:15

kunal642 mentioned this pull request Jan 4, 2018

[CARBONDATA-1978] Handled preaggregate issues with hive metastore #1758

Closed

5 tasks

rahulk2 force-pushed the pre_agg_issue branch from 31b50ef to d3b2818 Compare January 5, 2018 07:23

rahulk2 force-pushed the pre_agg_issue branch from d3b2818 to dfc6265 Compare January 5, 2018 12:17

rahulk2 force-pushed the pre_agg_issue branch from dfc6265 to 0b68e2f Compare January 7, 2018 08:22

rahulk2 force-pushed the pre_agg_issue branch from 0b68e2f to 3c6f6b9 Compare January 7, 2018 10:28

rahulk2 force-pushed the pre_agg_issue branch from 3c6f6b9 to 6dd1933 Compare January 7, 2018 14:24

rahulk2 force-pushed the pre_agg_issue branch from 6dd1933 to c50503b Compare January 8, 2018 11:11

[CARBONDATA-1954] CarbonHiveMetastore updated while dropping the pre-…

0d6b3b5

…aggregate table & code refactored to process metadata only if processMetadata() is called for preaggregate table

rahulk2 force-pushed the pre_agg_issue branch from c50503b to 0d6b3b5 Compare January 8, 2018 12:17

asfgit closed this in d33d347 Jan 8, 2018

xubo245 reviewed Jan 25, 2018

View reviewed changes

[CARBONDATA-1954] [Pre-Aggregate] CarbonHiveMetastore updated while dropping the Pre-Aggregate table & code refactored #1743

[CARBONDATA-1954] [Pre-Aggregate] CarbonHiveMetastore updated while dropping the Pre-Aggregate table & code refactored #1743

Conversation

rahulk2 commented Dec 29, 2017 • edited

ravipesala commented Dec 29, 2017

CarbonDataQA commented Dec 29, 2017

CarbonDataQA commented Dec 29, 2017

rahulk2 commented Jan 2, 2018

ravipesala commented Jan 2, 2018

gvramana Jan 3, 2018

Choose a reason for hiding this comment

gvramana Jan 3, 2018 • edited

Choose a reason for hiding this comment

gvramana Jan 3, 2018

Choose a reason for hiding this comment

gvramana commented Jan 3, 2018

CarbonDataQA commented Jan 3, 2018

CarbonDataQA commented Jan 3, 2018

ravipesala commented Jan 3, 2018

jackylk Jan 4, 2018 • edited

Choose a reason for hiding this comment

rahulk2 Jan 4, 2018

Choose a reason for hiding this comment

jackylk Jan 4, 2018

Choose a reason for hiding this comment

rahulk2 Jan 8, 2018

Choose a reason for hiding this comment

CarbonDataQA commented Jan 4, 2018

CarbonDataQA commented Jan 4, 2018

ravipesala commented Jan 4, 2018

ravipesala commented Jan 5, 2018

rahulk2 commented Jan 5, 2018

ravipesala commented Jan 5, 2018

CarbonDataQA commented Jan 5, 2018

CarbonDataQA commented Jan 5, 2018

ravipesala commented Jan 5, 2018

CarbonDataQA commented Jan 5, 2018

ravipesala commented Jan 7, 2018

CarbonDataQA commented Jan 7, 2018

CarbonDataQA commented Jan 7, 2018

ravipesala commented Jan 7, 2018

CarbonDataQA commented Jan 7, 2018

CarbonDataQA commented Jan 7, 2018

ravipesala commented Jan 7, 2018

CarbonDataQA commented Jan 7, 2018

CarbonDataQA commented Jan 7, 2018

CarbonDataQA commented Jan 8, 2018

ravipesala commented Jan 8, 2018

CarbonDataQA commented Jan 8, 2018

ravipesala commented Jan 8, 2018

ravipesala commented Jan 8, 2018

CarbonDataQA commented Jan 8, 2018

CarbonDataQA commented Jan 8, 2018

xubo245 Jan 25, 2018

Choose a reason for hiding this comment

rahulk2 commented Dec 29, 2017 •

edited

gvramana Jan 3, 2018 •

edited

jackylk Jan 4, 2018 •

edited