-
Notifications
You must be signed in to change notification settings - Fork 682
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Incorrect Repeated Field Schema Inference #1681
Comments
tustvold
added a commit
to tustvold/arrow-rs
that referenced
this issue
May 9, 2022
Don't treat embedded arrow schema as authoritative (apache#1663) Fix projection of nested parquet files (apache#1652) (apache#1654) Fix schema inference for repeated fields (apache#1681) Support reading alternative list representations from parquet (apache#1680)
tustvold
added a commit
to tustvold/arrow-rs
that referenced
this issue
May 9, 2022
Don't treat embedded arrow schema as authoritative (apache#1663) Fix projection of nested parquet files (apache#1652) (apache#1654) Fix schema inference for repeated fields (apache#1681) Support reading alternative list representations from parquet (apache#1680) Consistent handling of unsupported arrow types in parquet (apache#1666)
tustvold
added a commit
to tustvold/arrow-rs
that referenced
this issue
May 11, 2022
Don't treat embedded arrow schema as authoritative (apache#1663) Fix projection of nested parquet files (apache#1652) (apache#1654) Fix schema inference for repeated fields (apache#1681) Support reading alternative list representations from parquet (apache#1680)
alamb
pushed a commit
that referenced
this issue
May 13, 2022
* Separate parquet -> arrow conversion logic (#1655) Don't treat embedded arrow schema as authoritative (#1663) Fix projection of nested parquet files (#1652) (#1654) Fix schema inference for repeated fields (#1681) Support reading alternative list representations from parquet (#1680) * Add more tests * Pass pointers by reference * More docs * Fix lint * Review feedback * Review feedback * Fix test failures related to #1697
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
The schema inference logic in parquet does not infer the correct nullability for nested types.
For example
Will infer innerGroup and outerGroup as nullable lists with nullable elements, when they are neither.
To Reproduce
See test
Expected behavior
The nullability should be inferred correctly
Additional context
This has likely been hidden by the lack of support for repeated fields - #1680
The text was updated successfully, but these errors were encountered: