New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ARROW-11741: [C++] Fix decimal casts on big endian platforms #9554
Conversation
cc @kiszk |
Green Travis-CI build: https://travis-ci.com/github/pitrou/arrow/builds/217951780 |
Another possible fix would be to reorder Decimal fields based on endianness. That would only work for Decimal128, though (Decimal256 is based on a |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it makes most sense to reorder BasicDecimal128::{low_bits_, high_bits_}
based on endianness. Furthermore, BasicDecimal256 should (eventually) follow suit by replacing little_endian_array_
with native_endian_array_
.
Note that one issue with the |
When would any of these elements not be aligned to a 16 byte boundary? |
That would be quite unlikely indeed :-) |
Isn't it impossible for conformant data? |
No, data can be backed by a buffer that wasn't allocated by Arrow and that's not aligned naturally. That said, it's certainly extremely unlikely to happen in practice, and I don't think we actually care in other places, so I'm fine with ignoring that possibility. |
As far as Decimal256 is concerned, though, adapting the code may be non-trivial to conform to a native-endian layout. And if we do it only for Decimal128, we get less orthogonality. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nothing else in the compute namespace assumes data may be unaligned, so supporting it only here is suspect to me. We should at least have warnings that buffers are unaligned; even on platforms where unaligned access is not an error it does decrease performance.
I've added https://issues.apache.org/jira/browse/ARROW-11748 to track refactoring Decimal* into native-endian ordering.
In the meantime, let's merge this with a slight modification to make the intent clearer:
fe7aa32
to
d1f16aa
Compare
I applied your suggestions @bkietz , will merge. Thank you for the review! |
Thank you, I will watch ttps://issues.apache.org/jira/browse/ARROW-11748 |
Closes apache#9554 from pitrou/ARROW-11741-cast-decimal-big-endian Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
Closes apache#9554 from pitrou/ARROW-11741-cast-decimal-big-endian Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
No description provided.