fix(parquet/encoding): validate delta binary packed headers - #1012
Conversation
799a0fa to
d178c50
Compare
|
This has gone stale — #1027 landed on The header-validation half is still valuable and I would like to take it. On Could you rebase onto |
b9fa1a5 to
6fdfee2
Compare
zeroshade
left a comment
There was a problem hiding this comment.
Thanks for the rebase — this is exactly the right scope now. I confirmed the encoder Put() wraparound and its encoding_test.go case are gone, since #1027 already covers that on main, leaving just the header validation here.
Re-verified the remaining changes against current main: blockSize == 0 and an unbounded miniBlocksPerBlock were both accepted, and deltaBitWidths.Resize(int(d.miniBlocksPerBlock)) on an attacker-controlled count is a genuine allocation hazard on untrusted files. The block-size/miniblock divisibility rules, the values-per-miniblock multiple-of-32 check, and the INT32 range checks on the first value and min delta all match the Parquet spec, and I could not construct a valid input that any of them reject. Keeping the used-bit-width check at use time is the right call so unused padding miniblocks stay compatible. LGTM.
Rationale for this change
The delta binary packed decoder accepts header combinations that do not satisfy the Parquet encoding rules. Some combinations narrow into smaller Go fields or reach decoding with invalid miniblock geometry or integer widths.
What changes are included in this PR?
Are these changes tested?
Yes. Focused tests cover every header invariant, integer range, and per-type width. The full encoding and Parquet file suites pass.
Are there any user-facing changes?
INT32 files written by older arrow-go versions with non-conformant 33-bit miniblocks will now fail to read. This is intentional because those files violate the Parquet encoding rules and are also rejected by other Parquet implementations.