Use sorted index based filtering only for dictionary encoded column #6288
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
only for sorted column with dictionary
mcvsubbu
approved these changes
Nov 25, 2020
Codecov Report
@@ Coverage Diff @@
## master #6288 +/- ##
==========================================
+ Coverage 66.44% 74.01% +7.56%
==========================================
Files 1075 1252 +177
Lines 54773 61203 +6430
Branches 8168 8864 +696
==========================================
+ Hits 36396 45300 +8904
+ Misses 15700 12988 -2712
- Partials 2677 2915 +238
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
Jackie-Jiang
approved these changes
Nov 25, 2020
There was a problem hiding this comment.
(Optional) IMO it is cleaner if we check the dictionary when initializing the data source metadata since we don't support raw sorted index. In ImmutableDataSource.java, you can change the constructor of ImmutableDataSourceMetadata to _sorted = columnMetadata.isSorted() && columnMetadata.hasDictionary(). When we support raw sorted index in the future, we can change it back. Wdyt?
| Predicate.Type predicateType = predicateEvaluator.getPredicateType(); | ||
| if (predicateType == Predicate.Type.RANGE) { | ||
| if (dataSource.getDataSourceMetadata().isSorted()) { | ||
| if (dataSource.getDataSourceMetadata().isSorted() && (dataSource.getDictionary() != null)) { |
There was a problem hiding this comment.
(nit)
Suggested change
| if (dataSource.getDataSourceMetadata().isSorted() && (dataSource.getDictionary() != null)) { | |
| if (dataSource.getDataSourceMetadata().isSorted() && dataSource.getDictionary() != null) { |
| @@ -59,7 +65,7 @@ public static BaseFilterOperator getLeafFilterOperator(PredicateEvaluator predic | |||
| } else if (predicateType == Predicate.Type.REGEXP_LIKE) { | |||
| return new ScanBasedFilterOperator(predicateEvaluator, dataSource, numDocs); | |||
| } else { | |||
| if (dataSource.getDataSourceMetadata().isSorted()) { | |||
| if (dataSource.getDataSourceMetadata().isSorted() && (dataSource.getDictionary() != null)) { | |||
There was a problem hiding this comment.
(nit)
Suggested change
| if (dataSource.getDataSourceMetadata().isSorted() && (dataSource.getDictionary() != null)) { | |
| if (dataSource.getDataSourceMetadata().isSorted() && dataSource.getDictionary() != null) { |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently we build sorted index only if the column is dictionary encoded. However, when we write
isSortedin on-disk segment metadata, we write on the basis of pre-index stats collector. So, for a sorted column without dictionary, segment metadata will still indicate column as sortedproperties.setProperty(getKeyFor(column, IS_SORTED), String.valueOf(columnIndexCreationInfo.isSorted()));During query processing, when we create filter operator, we check the data source metadata to see if the column is sorted and create sorted index based filter operator. However, using this operator for any sorted raw column will lead to the following error stack since we end up using a raw value based predicate evaluator for a dictionary based filter operator.
The solution is to do the additional check on data source to see if the column is dictionary encoded or not