PARQUET-2021: Add description of RLE/bit-packed encoder #170
Conversation
This commit adds a new page with a detailed description of the RLE-bipacked hybrid encoder.
|
Can you word-wrap lines to ~80 characters? That will make reviewing a bit easier (and editing later as well). |
Good point. Sorry about that. I've now wrapped around 80 characters. |
pitrou
changed the title
rle-bitpacked.md
Outdated
| and thus the next run is RLE-encoded. We compute the number of repetitions | ||
| via `repetitions = h1 >> 1 = 8`. The body size is `body_length = ceil(bit_width, 8) = 1`, | ||
| which is consistent with `length = 5 = 1 + 2 + 1 + 1`. Since the body is | ||
| `0b00000001 = 1`, we conclude that this run is the number 1 repeated 8 times, |
There was a problem hiding this comment.
Is it? h1 >> 1 is the number of repetitions divided by 8, according to the grammar.
There was a problem hiding this comment.
Isn't only the bitpacked that is a multiple of 8?
rle-run := <rle-header> <repeated-value>
rle-header := varint-encode( (rle-run-len) << 1)
rle-run-len := *see 3 below*
|
I have not forgotten this. I am just making sure that this logic is right by performing different round-trips between Rust's parquet, parquet2 and pyarrow, with some control over the encodings. |
jorgecarleitao
changed the title
Addressed comments. Also removed the length of the stream; the length of the stream being declared in the beginning of the stream is V1 Page-specific; V2 pages use a different mechanism to declare the length of the stream.
|
Ok, I have resolved most comments except two that I have questions. I have also removed the "length of the stream" part, as that is v1 page-specific; v2 pages have a different way of specifying the length of the stream. I think we should leave that to a description of pages and what their buffers contain. |
|
@pitrou @julienledem , gentle ping :) |
|
Sorry @jorgecarleitao , I'll try to take a look in the next weeks. |
|
Hey @jorgecarleitao this is great work. Sorry I didn't see your PR |
|
@isnotinvain does it look good to you? |
|
also @toddlipcon designed this I believe |
This PR adds a new page with a detailed description of the RLE-bipacked hybrid encoder.
I had to implement this recently from scratch and found the current documentation a bit cryptic. As such, decided to spend some time to do a write up of how the encoder works in detail, which hopefully helps at reducing the barrier to implement this non-trivial encoding strategy.
NOTE: this is the first time I am contributing to parquet and have basic understanding of the format. I was hoping to have someone experienced in this encoding to review this.
I only described the decoding part. The encoding can be done in a follow-up PR.
The todo below will be fulfilled if there is consensus that this is useful.
Jira