Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[wip][CARBONDATA-2740]flat folder handling for implicit column and other bug fix #2504

Closed
wants to merge 1 commit into from
Closed

[wip][CARBONDATA-2740]flat folder handling for implicit column and other bug fix #2504

wants to merge 1 commit into from

Conversation

akashrn5
Copy link
Contributor

Problem

  1. When flat folder is enabled for table, pruning is not happening for implicit column.
  2. error message is wrong when complex column which does not have any string datatype column is given in local dictionary and exclude column.
  3. when data load is failed, corresponding segment file is not getting deleted.

Solution:

  1. when flat folder is enabled, based on the segment number present in the carbondata file name, pruning is taken care
  2. error message is corrected
  3. when data load is failed, segment file will be deleted

KanakaKumar
KanakaKumar reviewed Jul 13, 2018
View reviewed changes
integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonRelation.scala Outdated
datamaps.asScala.foreach { datamap =>
datamapsDirectorySize = datamapsDirectorySize + FileFactory
.getDirectorySize(
carbonTable.getTablePath + datamap.getDataMapSchema.getDataMapName)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use a utility method to make to dm path like org.apache.carbondata.core.util.path.CarbonTablePath#getDataMapStorePath.

@ravipesala
Copy link
Contributor

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5825/

@ravipesala
Copy link
Contributor

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5826/

@akashrn5 akashrn5 changed the title [CARBONDATA-2740]flat folder handling for implicit column and other bug fix [wip][CARBONDATA-2740]flat folder handling for implicit column and other bug fix Jul 13, 2018
@ravipesala
Copy link
Contributor

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5834/

@ravipesala
Copy link
Contributor

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5837/

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7169/

@CarbonDataQA
Copy link

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5945/

@akashrn5 akashrn5 closed this Jul 17, 2018
@akashrn5
Copy link
Contributor Author

please refer #2514 , as some chnages are moved to that PR, and flat folder support for implicit column will be raised in another PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@KanakaKumar KanakaKumar KanakaKumar left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@akashrn5 @ravipesala @CarbonDataQA @KanakaKumar