Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BitShuffle#shuffle for arrays of primitive datatypes does not consider the byte-order (leading to wrong results) #297

Open
foosmate opened this issue Nov 10, 2021 · 1 comment

Comments

@foosmate
Copy link

The underlying BitShuffle library works internally with an algorithm that was designed for the little-endian format. For this reason, the input data must always be passed in little-endian format.

However, the snappy-java wrapper directly passes an int[] array with the native byte order. Thus the Java implementation does not give the same results on little-endian and big-endian platforms for the shuffle / unshuffle methods.

The shuffle/unshuffle methods still work bijective so this issue is not easily noticeable.

How to fix:

The data array based on primitive types (e.g. int[], short[], long[], etc.) must be converted to little-endian format in the Java implementation if executed on big-endian platforms (before passing the buffer to the external library).

@xerial
Copy link
Owner

xerial commented May 23, 2023

Related #443

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants