Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Fixed error in writing ListArray to parquet #984

Merged
merged 2 commits into from
May 7, 2022
Merged

Conversation

jorgecarleitao
Copy link
Owner

When writing optional values to a required field, we must be mindful about the field nullability. Likewise, when writing required values to an optional field, we must ignore validity. Not doing this results in an incorrect file. This PR makes the implementation aware of the field nullability.

Closes #983 and #977

@jorgecarleitao jorgecarleitao added the bug Something isn't working label May 7, 2022
@jorgecarleitao jorgecarleitao changed the title Fixed error in writing List arrays to parquet Fixed error in writing ListArray to parquet May 7, 2022
@codecov
Copy link

codecov bot commented May 7, 2022

Codecov Report

Merging #984 (10a2006) into main (7d0908f) will decrease coverage by 0.05%.
The diff coverage is 67.00%.

@@            Coverage Diff             @@
##             main     #984      +/-   ##
==========================================
- Coverage   71.40%   71.35%   -0.06%     
==========================================
  Files         357      357              
  Lines       19728    19770      +42     
==========================================
+ Hits        14086    14106      +20     
- Misses       5642     5664      +22     
Impacted Files Coverage Δ
src/io/parquet/write/dictionary.rs 63.06% <44.44%> (+0.56%) ⬆️
src/io/parquet/write/mod.rs 58.62% <57.35%> (+0.11%) ⬆️
src/io/parquet/write/levels.rs 80.00% <66.21%> (-17.06%) ⬇️
src/io/parquet/write/row_group.rs 84.84% <77.77%> (-0.45%) ⬇️
src/io/parquet/write/binary/basic.rs 56.17% <100.00%> (ø)
src/io/parquet/write/binary/nested.rs 86.36% <100.00%> (ø)
src/io/parquet/write/boolean/basic.rs 97.50% <100.00%> (ø)
src/io/parquet/write/boolean/nested.rs 86.36% <100.00%> (ø)
src/io/parquet/write/fixed_len_bytes.rs 100.00% <100.00%> (ø)
src/io/parquet/write/primitive/basic.rs 84.21% <100.00%> (ø)
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7d0908f...10a2006. Read the comment docs.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parquet writing an empty list reads a null
1 participant