ARROW-6933: [Java] Suppor linear dictionary encoder #5692
Conversation
|
@emkornfield This is almost the same as #5058. Would you please take a look? |
|
|
||
| /** | ||
| * The dictionary for encoding/decoding. | ||
| * It must be sorted. |
There was a problem hiding this comment.
I wouldn't think it needs to be sorted? or at least clarify sorting.
There was a problem hiding this comment.
Sorry for the mistake. This sentence is removed.
|
|
||
| /** | ||
| * Constructs a dictionary encoder. | ||
| * @param dictionary the dictionary. Its entries should be sorted in the non-increasing order of their frequency. |
There was a problem hiding this comment.
sorting is performance not correctness correct?
There was a problem hiding this comment.
Good point. This is made explicity in the revised code.
| private Range range; | ||
|
|
||
| /** | ||
| * Constructs a dictionary encoder. |
There was a problem hiding this comment.
note that dictionary encoding is false by default.
There was a problem hiding this comment.
Good point. This is made explicit in the revised code.
Codecov Report
@@ Coverage Diff @@
## master #5692 +/- ##
==========================================
+ Coverage 88.93% 89.98% +1.04%
==========================================
Files 989 739 -250
Lines 134508 114814 -19694
Branches 1501 0 -1501
==========================================
- Hits 119627 103311 -16316
+ Misses 14516 11503 -3013
+ Partials 365 0 -365
Continue to review full report at Codecov.
|
For many scenarios, the distribution of dictionary entries is highly skewed. In other words, a few dictionary entries occurs much more frequently than others. If we can sort the dictionary by the non-increasing order of entry frequencies, and compare each value to encode from the beginning of the dictionary, we get the following benefits:
This is the basic idea behind the linear dictionary encoder. When the scenario is right (highly skewed dictionary distribution), it outperforms both search based encoder and hash table based encoders.