-
Notifications
You must be signed in to change notification settings - Fork 703
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[HOTFIX] Exclude filter doesn't work in presto carbon in cluster #3060
Conversation
@ravipesala : please check. |
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2243/ |
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10500/ |
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2462/ |
Is there test case to test this scene (use 'exclude filter' in presto carbon ), if not, better to add test case, so that other's changes will not affect this feature. |
@qiuchenjian : Test cases are there, But problem comes only in the cluster. Not in local environment due to jar dependency. |
LGTM |
b3041ad
to
11237eb
Compare
@ravipesala : local dictionary reset issue also, I have pushed in same PR. please check again |
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2293/ |
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10552/ |
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2513/ |
LGTM |
problem1: Exclude filter fails in cluster for presto carbon with exception. java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringBitmap at org.apache.carbondata.core.scan.filter.FilterUtil.prepareExcludeFilterMembers(FilterUtil.java:826) at org.apache.carbondata.core.scan.filter.FilterUtil.getDimColumnFilterInfoAfterApplyingCBO(FilterUtil.java:776) at org.apache.carbondata.core.scan.filter.FilterUtil.getFilterListForAllValues(FilterUtil.java:884) cause: RoaringBitmap jar is not added in the dependency, hence it is not present in the presto snapshot folder. solution : include RoaringBitmap in dependency. problem2: Local dictionary reset was not proper in vector for presto slice reader. cause : For each batch local dictionary was resetting. Instead of the actual place reset. solution : reset local dictionary when dictionary is set to null. This closes #3060
problem1: Exclude filter fails in cluster for presto carbon with exception. java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringBitmap at org.apache.carbondata.core.scan.filter.FilterUtil.prepareExcludeFilterMembers(FilterUtil.java:826) at org.apache.carbondata.core.scan.filter.FilterUtil.getDimColumnFilterInfoAfterApplyingCBO(FilterUtil.java:776) at org.apache.carbondata.core.scan.filter.FilterUtil.getFilterListForAllValues(FilterUtil.java:884) cause: RoaringBitmap jar is not added in the dependency, hence it is not present in the presto snapshot folder. solution : include RoaringBitmap in dependency. problem2: Local dictionary reset was not proper in vector for presto slice reader. cause : For each batch local dictionary was resetting. Instead of the actual place reset. solution : reset local dictionary when dictionary is set to null. This closes apache#3060
problem1: Exclude filter fails in cluster for presto carbon with exception.
cause: RoaringBitmap jar is not added in the dependency, hence it is not present in the presto snapshot folder.
solution : include RoaringBitmap in dependency.
problem2: Local dictionary reset was not proper in vector for presto slice reader.
cause : For each batch local dictionary was resetting. Instead of the actual place reset.
solution : reset local dictionary when dictionary is set to null.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
Any interfaces changed? NA
Any backward compatibility impacted? NA
Document update required? NA
Testing done. please find the report
Before:
After: