ARROW-6484: [Java] Enable create indexType for DictionaryEncoding according to dictionary value count #5321
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ctionary value count
|
+1, thank ou. |
pprudhvi
pushed a commit
to pprudhvi/arrow
that referenced
this pull request
Sep 16, 2019
…ording to dictionary value count Related to [ARROW-6484](https://issues.apache.org/jira/browse/ARROW-6484). Currently, when create DictionaryEncoding, we need to specify indexType, and it use Int(32, true) as default if this value is null. Actually, when dictionary valueCount is small, we should use Int(8,true)/Int(16,true) instead to reduce memory allocation. This issue is about to provide API for creating indexType according to valueCount and apply it to avro adapter for enum type. Closes apache#5321 from tianchen92/ARROW-6484 and squashes the following commits: fc1931b <tianchen> Enable create indexType for DictionaryEncoding according to dictionary value count Authored-by: tianchen <niki.lj@alibaba-inc.com> Signed-off-by: Micah Kornfield <emkornfield@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to ARROW-6484.
Currently, when create DictionaryEncoding, we need to specify indexType, and it use Int(32, true) as default if this value is null.
Actually, when dictionary valueCount is small, we should use Int(8,true)/Int(16,true) instead to reduce memory allocation.
This issue is about to provide API for creating indexType according to valueCount and apply it to avro adapter for enum type.