Describe the bug
parquet-format GH-607 adds clarification that in addition to unknown logical type enum variants, implementations must also handle unknown combinations of physical type and logical type (e.g. an INT32 column annotated with UUID). This crate currently checks for unsupported combinations and errors when encountering them.
To Reproduce
Try reading the file added in apache/parquet-testing#122 (parquet-testing/data/int32_with_uuid_logical_type.parquet).
Expected behavior
Unknown combinations should now be treated as having an unknown logical type i.e. on read they should be provided to users as the physical type with no type annotation, and statistics should be ignored. It's not clear (to me) what should be done on round trip.
It might be good to also log a warning when an unknown type is encountered to let users know of the degraded experience.
Additional context
We could continue to error on unknown combination, or add a gate to control behavior upon encountering unknown types.
Describe the bug
parquet-format GH-607 adds clarification that in addition to unknown logical type enum variants, implementations must also handle unknown combinations of physical type and logical type (e.g. an
INT32column annotated withUUID). This crate currently checks for unsupported combinations and errors when encountering them.To Reproduce
Try reading the file added in apache/parquet-testing#122 (parquet-testing/data/int32_with_uuid_logical_type.parquet).
Expected behavior
Unknown combinations should now be treated as having an unknown logical type i.e. on read they should be provided to users as the physical type with no type annotation, and statistics should be ignored. It's not clear (to me) what should be done on round trip.
It might be good to also log a warning when an unknown type is encountered to let users know of the degraded experience.
Additional context
We could continue to error on unknown combination, or add a gate to control behavior upon encountering unknown types.