PARQUET-2362: Clarify parquet encoding

[letian-jiang]

Dictionary page format: the entries in the dictionary - in dictionary order - using the plain encoding.

The dictionary entries are not sorted (or at least not always sorted).

There is no padding between values (except for the last byte) which is padded with 0s.

Minor change.

Jira

• https://issues.apache.org/jira/browse/PARQUET-2362

Commits

• My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[mapleFU]

Would you mind first create an issue like: https://issues.apache.org/jira/browse/PARQUET-2299 or use MINOR?

[mapleFU]

There is no padding between values (except for the last byte) which is padded with 0s.

This change looks good to me.

Dictionary page format: the entries in the dictionary - in dictionary order - using the plain encoding.

Does this means value in data page is same as position in dictionary page? 🤔

Also cc @wgtmac @gszadovszky

[letian-jiang]

Would you mind first create an issue like: https://issues.apache.org/jira/browse/PARQUET-2299 or use MINOR?

I will create a related issue once my JIRA account request is approved.

Does this means value in data page is same as position in dictionary page? 🤔

I think so. The data page contains dictionary code (i.e. offset in dictionary page )

[gszadovszky]

Maybe LSB first is not descriptive enough but at least we shall not remove it. What it wanted to say that for BOOLEANs it stores the bits from the least significant bit to the most significant one, instead of what is specified for BIT_PACKED. It practically means that the BOOLEAN bits are just written from left to right.

[JFinis]

IMHO, LSB first is pretty understandable and leaving it out makes the spec ambiguous.

Maybe it's only how I read it, but for me, LSB first pretty clearly conveys that "the first value will be stored in the least significant bit", so If I have the byte 0b00000001, this byte encodes 8 values, with the first one being 1 and all others being 0.

Maybe it helps spelling out ot more clearly, e.g.,

LSB first, i.e., the first value is stored in the least significant bit.

but for someone writing bit fiddling logic - which someone writing an encoder/decoder has to do - LSB first should be descriptive enough on its own IMHO.

[letian-jiang]

TIL. Thanks a lot!

[letian-jiang]

Made some updates. Please take another look. @mapleFU @gszadovszky @JFinis