Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect Repeated Field Schema Inference #1681

Closed
tustvold opened this issue May 9, 2022 · 0 comments · Fixed by #1682
Closed

Incorrect Repeated Field Schema Inference #1681

tustvold opened this issue May 9, 2022 · 0 comments · Fixed by #1682
Labels
bug parquet Changes to the parquet crate

Comments

@tustvold
Copy link
Contributor

tustvold commented May 9, 2022

Describe the bug

The schema inference logic in parquet does not infer the correct nullability for nested types.

For example

let message_type = "
message test_schema {
  OPTIONAL INT32 leaf1;
  REPEATED GROUP outerGroup {
    OPTIONAL INT32 leaf2;
    REPEATED GROUP innerGroup {
      OPTIONAL INT32 leaf3;
    }
  }
}
";
let parquet_group_type = parse_message_type(message_type).unwrap();
let parquet_schema = SchemaDescriptor::new(Arc::new(parquet_group_type));
let converted_arrow_schema =
parquet_to_arrow_schema(&parquet_schema, None).unwrap();

Will infer innerGroup and outerGroup as nullable lists with nullable elements, when they are neither.

To Reproduce

See test

Expected behavior

The nullability should be inferred correctly

Additional context

This has likely been hidden by the lack of support for repeated fields - #1680

@tustvold tustvold added the bug label May 9, 2022
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 9, 2022
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)

Fix schema inference for repeated fields (apache#1681)

Support reading alternative list representations from parquet (apache#1680)
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 9, 2022
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)

Fix schema inference for repeated fields (apache#1681)

Support reading alternative list representations from parquet (apache#1680)

Consistent handling of unsupported arrow types in parquet (apache#1666)
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 11, 2022
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)

Fix schema inference for repeated fields (apache#1681)

Support reading alternative list representations from parquet (apache#1680)
alamb pushed a commit that referenced this issue May 13, 2022
* Separate parquet -> arrow conversion logic (#1655)

Don't treat embedded arrow schema as authoritative (#1663)

Fix projection of nested parquet files (#1652) (#1654)

Fix schema inference for repeated fields (#1681)

Support reading alternative list representations from parquet (#1680)

* Add more tests

* Pass pointers by reference

* More docs

* Fix lint

* Review feedback

* Review feedback

* Fix test failures related to #1697
@alamb alamb added the parquet Changes to the parquet crate label May 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug parquet Changes to the parquet crate
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants