New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CARBONDATA-2819] Fixed cannot drop preagg datamap on table if the table has other type datamaps #2619

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
4 participants
@Sssan520
Contributor

Sssan520 commented Aug 8, 2018

1.Use "SET carbon.datamap.visible.{dbName}.{mainTable}.{datamapName} = {true or false}",if datamap provider is 'preagg',block this configuration.
2.When create preagg datamap, now also create its datamap schema file in system folder.
3.Use command "show datamap on table {tableName}" will get its all datamaps from datamap schema files

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

  • Any interfaces changed?No

  • Any backward compatibility impacted? No

  • Document update required? No

  • Testing done
    Please provide details on
    - Whether new unit test cases have been added or why no new tests are required?
    - How it is tested? Please attach test report.
    - Is it a performance related change? Please attach the performance test report.
    - Any additional information to help reviewers in testing this change.

  • For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 8, 2018

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7833/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 8, 2018

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6558/

@ravipesala

This comment has been minimized.

Show comment
Hide comment
@ravipesala

ravipesala Aug 8, 2018

Contributor

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6207/

Contributor

ravipesala commented Aug 8, 2018

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6207/

@Sssan520

This comment has been minimized.

Show comment
Hide comment
@Sssan520

Sssan520 Aug 8, 2018

Contributor

retest this please

Contributor

Sssan520 commented Aug 8, 2018

retest this please

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 8, 2018

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7839/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 8, 2018

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6563/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 8, 2018

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7841/

@ravipesala

This comment has been minimized.

Show comment
Hide comment
@ravipesala

ravipesala Aug 8, 2018

Contributor

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6214/

Contributor

ravipesala commented Aug 8, 2018

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6214/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 8, 2018

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6565/

@Sssan520

This comment has been minimized.

Show comment
Hide comment
@Sssan520

Sssan520 Aug 8, 2018

Contributor

retest this please

Contributor

Sssan520 commented Aug 8, 2018

retest this please

@ravipesala

This comment has been minimized.

Show comment
Hide comment
@ravipesala

ravipesala Aug 8, 2018

Contributor

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6215/

Contributor

ravipesala commented Aug 8, 2018

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6215/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 8, 2018

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7842/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 8, 2018

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6567/

@Sssan520

This comment has been minimized.

Show comment
Hide comment
@Sssan520

Sssan520 Aug 9, 2018

Contributor

retest this please

Contributor

Sssan520 commented Aug 9, 2018

retest this please

@ravipesala

This comment has been minimized.

Show comment
Hide comment
@ravipesala

ravipesala Aug 9, 2018

Contributor

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6218/

Contributor

ravipesala commented Aug 9, 2018

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6218/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 9, 2018

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7847/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 9, 2018

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6572/

@Sssan520

This comment has been minimized.

Show comment
Hide comment
@Sssan520

Sssan520 Aug 9, 2018

Contributor

retest this please

Contributor

Sssan520 commented Aug 9, 2018

retest this please

@ravipesala

This comment has been minimized.

Show comment
Hide comment
@ravipesala

ravipesala Aug 9, 2018

Contributor

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6219/

Contributor

ravipesala commented Aug 9, 2018

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6219/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 9, 2018

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7848/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 9, 2018

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6573/

@Sssan520

This comment has been minimized.

Show comment
Hide comment
@Sssan520

Sssan520 Aug 9, 2018

Contributor

retest this please

Contributor

Sssan520 commented Aug 9, 2018

retest this please

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 9, 2018

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7851/

@xuchuanyin

please optimize the title and description of the PR

Show outdated Hide outdated core/src/main/java/org/apache/carbondata/core/util/SessionParams.java
@@ -175,7 +175,7 @@ object PreAggregateDataMapExample {
// create pre-aggregate table by datamap
spark.sql("""
CREATE datamap preagg_avg on table personTable using 'preaggregate' as
CREATE datamap preagg_avg_dm on table personTable using 'preaggregate' as

This comment has been minimized.

@xuchuanyin

xuchuanyin Aug 9, 2018

Contributor

why this modification is needed?

@xuchuanyin

xuchuanyin Aug 9, 2018

Contributor

why this modification is needed?

This comment has been minimized.

@Sssan520

Sssan520 Aug 10, 2018

Contributor

As datamap name is a global, there is already the same datamap name before.

@Sssan520

Sssan520 Aug 10, 2018

Contributor

As datamap name is a global, there is already the same datamap name before.

dropTableCommand = new CarbonDropTableCommand(
true,
new Some<>(dbName),
tableName,
true);
dropTableCommand.processMetadata(sparkSession);
DataMapStoreManager.getInstance().dropDataMapSchema(dataMapSchema.getDataMapName());

This comment has been minimized.

@xuchuanyin

xuchuanyin Aug 9, 2018

Contributor

Can you check how MV deal with this? In another word, what's the logic of dropping a MV?

@xuchuanyin

xuchuanyin Aug 9, 2018

Contributor

Can you check how MV deal with this? In another word, what's the logic of dropping a MV?

This comment has been minimized.

@Sssan520

Sssan520 Aug 10, 2018

Contributor

when create a mv datamap, will create a datamap scheme file and register a datamap catalog; then when drop the mv datamap,will delete the datamap schema file and unregister the datamap catalog.

@Sssan520

Sssan520 Aug 10, 2018

Contributor

when create a mv datamap, will create a datamap scheme file and register a datamap catalog; then when drop the mv datamap,will delete the datamap schema file and unregister the datamap catalog.

@@ -114,12 +114,6 @@ case class CarbonDropDataMapCommand(
LOGGER.audit(s"Deleting datamap [$dataMapName] under table [$tableName]")
// drop index datamap on the main table
if (mainTable != null &&

This comment has been minimized.

@xuchuanyin

xuchuanyin Aug 9, 2018

Contributor

Can you explain why this modification is needed?

@xuchuanyin

xuchuanyin Aug 9, 2018

Contributor

Can you explain why this modification is needed?

This comment has been minimized.

@Sssan520

Sssan520 Aug 10, 2018

Contributor

when drop preagg datamap , should delete the datamap schema file and main tableinfo schema info.

@Sssan520

Sssan520 Aug 10, 2018

Contributor

when drop preagg datamap , should delete the datamap schema file and main tableinfo schema info.

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 9, 2018

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6577/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 10, 2018

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6605/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 10, 2018

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7882/

@ravipesala

This comment has been minimized.

Show comment
Hide comment
@ravipesala

ravipesala Aug 10, 2018

Contributor

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6244/

Contributor

ravipesala commented Aug 10, 2018

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6244/

@Sssan520 Sssan520 changed the title from [CARBONDATA-2819] Fixed cannot drop preagg datamap on table if the tab… to [CARBONDATA-2819] Fixed cannot drop preagg datamap on table if the table has other type datamaps Aug 10, 2018

@Sssan520

This comment has been minimized.

Show comment
Hide comment
@Sssan520

Sssan520 Aug 10, 2018

Contributor

retest this please

Contributor

Sssan520 commented Aug 10, 2018

retest this please

@ravipesala

This comment has been minimized.

Show comment
Hide comment
@ravipesala

ravipesala Aug 10, 2018

Contributor

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6245/

Contributor

ravipesala commented Aug 10, 2018

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6245/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 10, 2018

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7883/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 10, 2018

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6607/

@Sssan520

This comment has been minimized.

Show comment
Hide comment
@Sssan520

Sssan520 Aug 14, 2018

Contributor

retest this please

Contributor

Sssan520 commented Aug 14, 2018

retest this please

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 14, 2018

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7901/

@CarbonDataQA

This comment has been minimized.

Show comment
Hide comment
@CarbonDataQA

CarbonDataQA commented Aug 14, 2018

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6625/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment