Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C++][Parquet] Support Crc32 write and verify for DICT_PAGE #34147

Closed
mapleFU opened this issue Feb 12, 2023 · 0 comments · Fixed by #34254
Closed

[C++][Parquet] Support Crc32 write and verify for DICT_PAGE #34147

mapleFU opened this issue Feb 12, 2023 · 0 comments · Fixed by #34254
Assignees
@mapleFU
Labels
Component: C++ Component: Parquet Type: enhancement
Milestone
12.0.0

Comments

@mapleFU
Copy link
Member

mapleFU commented Feb 12, 2023

Describe the enhancement requested

This issue is part of https://issues.apache.org/jira/browse/ARROW-17904 . Previously, we support crc32 for DATA_PAGE_V1, in this patch we need to support DICT_PAGE. The todo is:

  1. Support write and check DICT_PAGE
  2. Generate a checksum DICT_PAGE using parquet-mr, and put it into parquet-mr. Test reading in arrow parquet
  3. Testing DictPage checksum

Component(s)

C++, Parquet
@github-actions github-actions bot mentioned this issue Feb 18, 2023
Merged
3 tasks
wjones127 pushed a commit that referenced this issue Mar 9, 2023
@mapleFU
GH-34147: [C++][Parquet] Support crc count and checking on DICTIONARY…
0ac0f73 
…_PAGE (#34254)

### Rationale for this change

Support counting CRC for dictionary page

### What changes are included in this PR?

* [x] Support counting crc for dictionary page when crc enabled
* [x] Support verify crc for dictionary page when checking crc enabled
* [x] Generate CRC dictionary file using parquet-mr, and do cross-verification

### Are these changes tested?

Yes

### Are there any user-facing changes?

Yes, dictionary page could use crc after that

* Closes: #34147

Authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: Will Jones <willjones127@gmail.com>
@wjones127 wjones127 added this to the 12.0.0 milestone Mar 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mapleFU mapleFU

Labels
Component: C++ Component: Parquet Type: enhancement
Projects
None yet
Milestone
12.0.0
Development

Successfully merging a pull request may close this issue.

GH-34147: [C++][Parquet] Support crc count and checking on DICTIONARY_PAGE mapleFU/arrow
2 participants
@wjones127 @mapleFU