Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StructArrayReader Cannot Handle Nested Lists #1651

Closed
tustvold opened this issue May 5, 2022 · 0 comments · Fixed by #1700
Closed

StructArrayReader Cannot Handle Nested Lists #1651

tustvold opened this issue May 5, 2022 · 0 comments · Fixed by #1700
Assignees
Labels
bug parquet Changes to the parquet crate

Comments

@tustvold
Copy link
Contributor

tustvold commented May 5, 2022

Describe the bug

When reading the parquet file provided by @kesavkolla here the reader bails out with "Not all children array length are the same".

Interestingly if you just project column 32 i.e.

let mut reader = reader.get_record_reader_by_columns(vec![32], 1024).unwrap();

You get ParquetError("Arrow: item_reader rep levels are None.")

If you project 31 and 32 you get the above error.

To Reproduce

Run the test on the provided file

Expected behavior

The file should be read correctly

@tustvold tustvold added the bug label May 5, 2022
@tustvold tustvold self-assigned this May 5, 2022
@tustvold tustvold changed the title Parquet Reader Bails Out Reading Parquet File StructArrayReader Cannot Handle Nested Lists May 13, 2022
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 13, 2022
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 13, 2022
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 13, 2022
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 13, 2022
tustvold added a commit that referenced this issue May 19, 2022
* Fix StructArrayReader handling nested lists (#1651)

* Rename nulls to validity
@alamb alamb added the parquet Changes to the parquet crate label May 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug parquet Changes to the parquet crate
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants