Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CARBONDATA-1284]Implement hive based schema storage in carbon #1149

Closed
wants to merge 6 commits into from
Closed

[CARBONDATA-1284]Implement hive based schema storage in carbon #1149

wants to merge 6 commits into from

Conversation

ravipesala
Copy link
Contributor

@ravipesala ravipesala commented Jul 8, 2017 •
edited

Implemented hive based schema storage in carbon
By using the property spark.carbon.hive.schema.store to true , carbon can only save schema to hive metastore. it will not store in file system.

@CarbonDataQA
Copy link

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/373/

@CarbonDataQA
Copy link

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2961/

@CarbonDataQA
Copy link

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/374/

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2962/

@ravipesala ravipesala changed the title Implement hive based schema storage in carbon [WIP]Implement hive based schema storage in carbon Jul 8, 2017
@CarbonDataQA
Copy link

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/375/

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2963/

@CarbonDataQA
Copy link

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/391/

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2980/

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2987/

@CarbonDataQA
Copy link

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/398/

@ravipesala ravipesala changed the base branch from master to metadata July 10, 2017 08:34
@ravipesala ravipesala changed the title [WIP]Implement hive based schema storage in carbon [CARBONDATA-1284]Implement hive based schema storage in carbon Jul 10, 2017
@CarbonDataQA
Copy link

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/409/

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2998/

QiangCai
QiangCai reviewed Jul 11, 2017
View reviewed changes
integration/spark2/src/main/scala/org/apache/carbondata/spark/util/CarbonSparkUtil.scala Outdated
@@ -42,4 +44,14 @@ object CarbonSparkUtil {
}
CarbonMetaData(dimensionsAttr, measureAttr, carbonTable, DictionaryMap(dictionary.toMap))
}

def createCarbonRelation(tableInfo: TableInfo, tablePath: String): CarbonRelation = {
val identifier = AbsoluteTableIdentifier fromTablePath(tablePath)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AbsoluteTableIdentifier.fromTablePath(tablePath)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@CarbonDataQA
Copy link

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3019/

@CarbonDataQA
Copy link

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/431/

@jackylk
Copy link
Contributor

jackylk commented Jul 11, 2017

LGTM
Test case need to be solved in another PR

asfgit pushed a commit that referenced this pull request Jul 11, 2017
@ravipesala @jackylk
[CARBONDATA-1284]Implement hive based schema storage in carbon
2a9debf 
This closes #1149
@gvramana
Copy link
Contributor

@ravipesala Previously schema refresh is using store level timestamp file and then table schema file modification timestamp. How now refresh is detected?

@CarbonDataQA
Copy link

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/549/

@CarbonDataQA
Copy link

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3141/

@asfgit asfgit closed this in 9a8fac3 Jul 25, 2017
sraghunandan pushed a commit to sraghunandan/carbondata-1 that referenced this pull request Jul 26, 2017
@ravipesala @sraghunandan
[CARBONDATA-1284]Implement hive based schema storage in carbon
5f6956e 
This closes apache#1149
sraghunandan pushed a commit to sraghunandan/carbondata-1 that referenced this pull request Jul 27, 2017
@ravipesala @sraghunandan
[CARBONDATA-1284]Implement hive based schema storage in carbon
042a05a 
This closes apache#1149
anubhav100 pushed a commit to anubhav100/incubator-carbondata that referenced this pull request Jul 31, 2017
@ravipesala @anubhav100
[CARBONDATA-1284]Implement hive based schema storage in carbon
d3dfe2a 
This closes apache#1149
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@QiangCai QiangCai QiangCai left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@ravipesala @CarbonDataQA @jackylk @gvramana @QiangCai