-
Notifications
You must be signed in to change notification settings - Fork 727
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parquet v8.0.0 panics when reading all null column to NullArray #1245
Comments
Yup that definitely sounds plausible, I'll take a look shortly, thanks for reporting 👍 FWIW I notice it is using the NullArrayReader, I don't know if you can encourage pandas to give it a different column type (e.g. Int32), but that might be an interesting data points |
Ah, so it is. I gave it the encouragement, and it looks like that works. Interestingly, trying
vs.
|
I've found the bug, the derp is strong, and will only impact NullArray - will post a PR shortly 😄
Yeah the And yes, there is a base64 encoded flatbuffer, inside a thrift metadata payload, inside a parquet file - it's wild 😆 |
Thanks again for reporting, fix in #1246 |
Describe the bug
When reading a Parquet file with a single row, containing a single optional
int32
column containingnull
the parquet reader panics.This was found when trying to upgrade to Arrow/Parquet v8.0.0. The original file contained additional columns but was reduced to this.
The referenced branch contains a Parquet file containing a single nullable Int32 value. In this case I produced it using the following Pandas code:
The file loads correctly using
6.3.0
. With8.0.0
it panics.To Reproduce
Steps to reproduce the behavior:
Run the test from this branch.
https://github.com/bjchambers/arrow-rs/tree/repro-parquet-panic
https://github.com/bjchambers/arrow-rs/blob/repro-parquet-panic/parquet/src/arrow/arrow_reader.rs#L990
Expected behavior
Not a panic. The behavior in
6.3.0
was expected.Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: