Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C++][Parquet] Parquet Support CRC on DataPage V2 #35171

Closed
mapleFU opened this issue Apr 17, 2023 · 0 comments · Fixed by #35242
Closed

[C++][Parquet] Parquet Support CRC on DataPage V2 #35171

mapleFU opened this issue Apr 17, 2023 · 0 comments · Fixed by #35242
Assignees
@mapleFU
Labels
Component: C++ Component: Parquet Type: enhancement
Milestone
13.0.0

Comments

@mapleFU
Copy link
Member

mapleFU commented Apr 17, 2023

Describe the enhancement requested

Currently, we already support crc on DataPage V1 and Dictionary Page. Finally we need to support DataPage v2.
DataPage v2 is not used online, and parquet-testing not have data page v2 with crc, because parquet-mr didn't implement it. So we cannot use the file from parquet-testing.

Component(s)

C++, Parquet
@github-actions github-actions bot mentioned this issue Apr 20, 2023
Merged
wjones127 pushed a commit that referenced this issue May 1, 2023
@mapleFU
GH-35171: [C++][Parquet] Implement CRC for data page v2 (#35242)
99ac74e 
### Rationale for this change

Implement crc for data page v2.

### What changes are included in this PR?

1. When writing, able to write crc for data page v2
2. When reading, able to read crc for data page v2
3. Testing

### Are these changes tested?

Yes, but parquet-testing don't have crc page v2 test. So no integration test

### Are there any user-facing changes?

* Closes: #35171

Authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: Will Jones <willjones127@gmail.com>
@wjones127 wjones127 added this to the 13.0.0 milestone May 1, 2023
liujiacheng777 pushed a commit to LoongArch-Python/arrow that referenced this issue May 11, 2023
@mapleFU @liujiacheng777
apacheGH-35171: [C++][Parquet] Implement CRC for data page v2 (apache…
641f80b 
…#35242)

### Rationale for this change

Implement crc for data page v2.

### What changes are included in this PR?

1. When writing, able to write crc for data page v2
2. When reading, able to read crc for data page v2
3. Testing

### Are these changes tested?

Yes, but parquet-testing don't have crc page v2 test. So no integration test

### Are there any user-facing changes?

* Closes: apache#35171

Authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: Will Jones <willjones127@gmail.com>
ArgusLi pushed a commit to Bit-Quill/arrow that referenced this issue May 15, 2023
@mapleFU @ArgusLi
apacheGH-35171: [C++][Parquet] Implement CRC for data page v2 (apache…
e2467d0 
…#35242)

### Rationale for this change

Implement crc for data page v2.

### What changes are included in this PR?

1. When writing, able to write crc for data page v2
2. When reading, able to read crc for data page v2
3. Testing

### Are these changes tested?

Yes, but parquet-testing don't have crc page v2 test. So no integration test

### Are there any user-facing changes?

* Closes: apache#35171

Authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: Will Jones <willjones127@gmail.com>
rtpsw pushed a commit to rtpsw/arrow that referenced this issue May 16, 2023
@mapleFU @rtpsw
apacheGH-35171: [C++][Parquet] Implement CRC for data page v2 (apache…
9dc5d2c 
…#35242)

### Rationale for this change

Implement crc for data page v2.

### What changes are included in this PR?

1. When writing, able to write crc for data page v2
2. When reading, able to read crc for data page v2
3. Testing

### Are these changes tested?

Yes, but parquet-testing don't have crc page v2 test. So no integration test

### Are there any user-facing changes?

* Closes: apache#35171

Authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: Will Jones <willjones127@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mapleFU mapleFU

Labels
Component: C++ Component: Parquet Type: enhancement
Projects
None yet
Milestone
13.0.0
Development

Successfully merging a pull request may close this issue.

GH-35171: [C++][Parquet] Implement CRC for data page v2 mapleFU/arrow
2 participants
@wjones127 @mapleFU