[CARBONDATA-3353 ]Fixed MinMax Based Pruning for Measure column in case of Legacy store #3176
Conversation
Indhumathi27
changed the title
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2863/ |
...org/apache/carbondata/core/scan/filter/executer/RowLevelRangeLessThanFilterExecuterImpl.java
Show resolved
Hide resolved
|
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/11123/ |
|
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/3094/ |
|
@Indhumathi27 I think earlier we used to reverse the min and max in case of the legacy store but now we are completely ignoring the min/max of them. Please check references of reversing the min/max and remove them as we don't need it now. |
|
@ravipesala removed code for reversing the min/max in case of legacy store. Please review |
|
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/3111/ |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2880/ |
|
Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/11140/ |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2881/ |
|
retest this please |
|
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/11141/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2883/ |
|
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/3112/ |
|
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/11143/ |
|
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/3114/ |
|
LGTM |
…se of Legacy store Why this PR needed? Problem: For table created and loaded with legacy store having a measure column, while building the page min max, min is written as max and viceversa, so blocklet level minmax is wrong. With current version, when we query with filter on measure column, measure filter pruning is skipping some blocks and giving wrong results. Solution: Skip MinMax based pruning in case of legacy store for measure column. This closes #3176
…se of Legacy store Why this PR needed? Problem: For table created and loaded with legacy store having a measure column, while building the page min max, min is written as max and viceversa, so blocklet level minmax is wrong. With current version, when we query with filter on measure column, measure filter pruning is skipping some blocks and giving wrong results. Solution: Skip MinMax based pruning in case of legacy store for measure column. This closes apache#3176
Why this PR needed?
Problem:
For table created and loaded with legacy store having a measure column, while building the page min max,
min is written as max and viceversa, so blocklet level minmax is wrong. With current version, when we query with filter on measure column, measure filter pruning is skipping some blocks and giving wrong results.
Solution:
Skip MinMax based pruning in case of legacy store for measure column.
Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
Manually tested
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.