-
Notifications
You must be signed in to change notification settings - Fork 717
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Construct decimal from i32 #5325
Comments
Seems in arrow-rs/parquet/src/data_type.rs Lines 371 to 379 in 7ba36b0
use big-endian byte order but from the parquet doc : https://github.com/apache/parquet-format/blob/eb4b31c1d64a01088d02a2f9aefc6c17c54cc6fc/Encodings.md?plain=1#L35-L42 Should use little-endian
@tustvold could you plz take a look at this, thx! |
Yes, this does not appear to be correct, but I am not very familiar with this code. It is part of the record API, and not used by the various arrow APIs. Perhaps @sunchao has some context here? |
Thanks for your reply, like you saying it seems only used on some record API testing, i will try to fix this without other concerns。 |
I think this code is for converting a I don't remember exactly why we chose big-endian instead of little-endian for the above (it was added in sunchao/parquet-rs#103). If one needs to writes out to Parquet after reading the decimals, then it might be better to use little-endian here to save some conversion cost. On the other hand, Java's |
Right, the part that confused me is the from_bytes constructor doesn't perform the same conversion. So it isn't clear whether the bytes stored are little or big endian... |
I recommend we at least put some of this knowledge into the code as comments, even if we don't change the actual implementation |
I think the input of This is also another reason for using big-endian for |
Aah, parquet itself is inconsistent 😭 Ok so the next step I think would be to document that the Decimal struct stores data big-endian encoded, and to highlight that if using i32 or i64 as the physical type, parquet uses little endian encoding including in statistics and bloom filters (as they're documented to mirror the PLAIN encoding), but uses big-endian encoding for byte-array based physical types, including in statistics and bloom filters. |
So, i assume https://github.com/apache/parquet-format/blob/eb4b31c1d64a01088d02a2f9aefc6c17c54cc6fc/Encodings.md?plain=1#L41 bytes_array' bytes not define endian-order , but why in decimal need force it big-endian https://github.com/apache/parquet-format/blob/eb4b31c1d64a01088d02a2f9aefc6c17c54cc6fc/LogicalTypes.md?plain=1#L214-L216 🤔 And i still not understand which situation need construct decimal from |
@Ted-Jiang I guess it is because on the Java side, decimal is typically represented using |
(the same applies to the other types here as well)
Originally posted by @alamb in apache/datafusion#8930 (comment)
The text was updated successfully, but these errors were encountered: