Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HOTFIX] Exclude filter doesn't work in presto carbon in cluster #3060

Closed
wants to merge 2 commits into from

Conversation

ajantha-bhat
Copy link
Member

@ajantha-bhat ajantha-bhat commented Jan 9, 2019

problem1: Exclude filter fails in cluster for presto carbon with exception.

java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringBitmap
        at org.apache.carbondata.core.scan.filter.FilterUtil.prepareExcludeFilterMembers(FilterUtil.java:826)
        at org.apache.carbondata.core.scan.filter.FilterUtil.getDimColumnFilterInfoAfterApplyingCBO(FilterUtil.java:776)
        at org.apache.carbondata.core.scan.filter.FilterUtil.getFilterListForAllValues(FilterUtil.java:884)

cause: RoaringBitmap jar is not added in the dependency, hence it is not present in the presto snapshot folder.
solution : include RoaringBitmap in dependency.

problem2: Local dictionary reset was not proper in vector for presto slice reader.
cause : For each batch local dictionary was resetting. Instead of the actual place reset.
solution : reset local dictionary when dictionary is set to null.

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

  • Any interfaces changed? NA

  • Any backward compatibility impacted? NA

  • Document update required? NA

  • Testing done. please find the report
    Before:

presto:default> select name from nbig where name < 'aj' limit 5;
Query 20190109_131447_00004_qhrfk failed: org/roaringbitmap/RoaringBitmap

After:

presto:default> select name from nbig where name < 'aj' limit 5;
  name  
--------
 208    
 150209 
 150210 
 150211 
 150212 
(5 rows)
  • For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA

@ajantha-bhat
Copy link
Member Author

@ravipesala : please check.

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2243/

@CarbonDataQA
Copy link

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10500/

@CarbonDataQA
Copy link

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2462/

@qiuchenjian
Copy link
Contributor

Is there test case to test this scene (use 'exclude filter' in presto carbon ), if not, better to add test case, so that other's changes will not affect this feature.

@ajantha-bhat
Copy link
Member Author

@qiuchenjian : Test cases are there, But problem comes only in the cluster. Not in local environment due to jar dependency.

@ravipesala
Copy link
Contributor

LGTM

@ajantha-bhat
Copy link
Member Author

@ravipesala : local dictionary reset issue also, I have pushed in same PR. please check again

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2293/

@CarbonDataQA
Copy link

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10552/

@CarbonDataQA
Copy link

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2513/

@ravipesala
Copy link
Contributor

LGTM

@asfgit asfgit closed this in 9608413 Jan 16, 2019
asfgit pushed a commit that referenced this pull request Jan 21, 2019
problem1: Exclude filter fails in cluster for presto carbon with exception.

java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringBitmap
        at org.apache.carbondata.core.scan.filter.FilterUtil.prepareExcludeFilterMembers(FilterUtil.java:826)
        at org.apache.carbondata.core.scan.filter.FilterUtil.getDimColumnFilterInfoAfterApplyingCBO(FilterUtil.java:776)
        at org.apache.carbondata.core.scan.filter.FilterUtil.getFilterListForAllValues(FilterUtil.java:884)

cause: RoaringBitmap jar is not added in the dependency, hence it is not present in the presto snapshot folder.
solution : include RoaringBitmap in dependency.

problem2: Local dictionary reset was not proper in vector for presto slice reader.
cause : For each batch local dictionary was resetting. Instead of the actual place reset.
solution : reset local dictionary when dictionary is set to null.

This closes #3060
qiuchenjian pushed a commit to qiuchenjian/carbondata that referenced this pull request Jun 14, 2019
problem1: Exclude filter fails in cluster for presto carbon with exception.

java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringBitmap
        at org.apache.carbondata.core.scan.filter.FilterUtil.prepareExcludeFilterMembers(FilterUtil.java:826)
        at org.apache.carbondata.core.scan.filter.FilterUtil.getDimColumnFilterInfoAfterApplyingCBO(FilterUtil.java:776)
        at org.apache.carbondata.core.scan.filter.FilterUtil.getFilterListForAllValues(FilterUtil.java:884)

cause: RoaringBitmap jar is not added in the dependency, hence it is not present in the presto snapshot folder.
solution : include RoaringBitmap in dependency.

problem2: Local dictionary reset was not proper in vector for presto slice reader.
cause : For each batch local dictionary was resetting. Instead of the actual place reset.
solution : reset local dictionary when dictionary is set to null.

This closes apache#3060
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants