Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

parquet_to_arrow_schema_by_columns Incorrectly Handles Nested Types #1654

Closed
tustvold opened this issue May 5, 2022 · 0 comments · Fixed by #1682
Closed

parquet_to_arrow_schema_by_columns Incorrectly Handles Nested Types #1654

tustvold opened this issue May 5, 2022 · 0 comments · Fixed by #1682
Assignees
@tustvold
Labels
bug parquet Changes to the parquet crate

Comments

@tustvold
Copy link
Contributor

tustvold commented May 5, 2022 •
edited
Loading

Describe the bug

parquet_to_arrow_schema_by_columns calls ArrowSchema::field_with_name with the name of the projected column from the parquet schema. In the case of a nested field, this will either fallback to deducing the type from the parquet schema, or potentially use the schema of a root-level field that happens to have the same name 😱

To Reproduce

Inspect code

Expected behavior

parquet_to_arrow_schema_by_columns should correctly handle arrow schema for nested types

Additional Context

Encountered whilst attempting to fix #1652 which in turn feeds into apache/datafusion#2439
@tustvold tustvold added the bug label May 5, 2022
@tustvold tustvold mentioned this issue May 5, 2022
Closed
@tustvold tustvold self-assigned this May 5, 2022
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 9, 2022
@tustvold
Separate parquet -> arrow conversion logic (apache#1655)
c744337 
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 9, 2022
@tustvold
Separate parquet -> arrow conversion logic (apache#1655)
4b98333 
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)

Fix schema inference for repeated fields (apache#1681)

Support reading alternative list representations from parquet (apache#1680)
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 9, 2022
@tustvold
Separate parquet -> arrow conversion logic (apache#1655)
6eb932c 
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)

Fix schema inference for repeated fields (apache#1681)

Support reading alternative list representations from parquet (apache#1680)

Consistent handling of unsupported arrow types in parquet (apache#1666)
@tustvold tustvold mentioned this issue May 9, 2022
Merged
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 11, 2022
@tustvold
Separate parquet -> arrow conversion logic (apache#1655)
e2f12de 
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)

Fix schema inference for repeated fields (apache#1681)

Support reading alternative list representations from parquet (apache#1680)
alamb pushed a commit that referenced this issue May 13, 2022
@tustvold
Fix Parquet Arrow Schema Inference (#1682)
5b154ea 
* Separate parquet -> arrow conversion logic (#1655)

Don't treat embedded arrow schema as authoritative (#1663)

Fix projection of nested parquet files (#1652) (#1654)

Fix schema inference for repeated fields (#1681)

Support reading alternative list representations from parquet (#1680)

* Add more tests

* Pass pointers by reference

* More docs

* Fix lint

* Review feedback

* Review feedback

* Fix test failures related to #1697
@alamb alamb added the parquet Changes to the parquet crate label May 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tustvold tustvold

Labels
bug parquet Changes to the parquet crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Parquet Reader's Arrow Schema Inference tustvold/arrow-rs
2 participants
@alamb @tustvold