[Java] Endianness field not emitted in IPC stream

[asfimport]

It seems the Java IPC writer implementation does not emit the Endianness information at all (making it Little by default). This complicates interoperability with the C++ IPC reader, which does read this information and acts on it to decide whether it needs to byteswap the incoming data.

Reporter: Antoine Pitrou / @pitrou
Assignee: Kazuaki Ishizaki / @kiszk

PRs and other links:

• GitHub Pull Request #12777

_{Note: This issue was originally created as ARROW-15778. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
The offending code seems to be there:
https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/types/pojo/Schema.java#L202-L213

This seems reasonably easy to fix (perhaps a one-line fix, though a test should ideally be added as well).

@emkornfield @kiszk

[asfimport]

Antoine Pitrou / @pitrou:
Also cc @lidavidm since this affects Flight.

[asfimport]

David Li / @lidavidm:
Flight doesn't do any endianness detection/negotiation anyways (it expects producer/consumer to set appropriate options) though we should eventually fix that.

[asfimport]

Antoine Pitrou / @pitrou:
I'm not sure any form of negotiation is needed? The way it works at the IPC level is that the writer emits data in whichever endianness it chooses (also setting the corresponding metadata field to the appropriate value) and the reader decides to byte-swap data is required. So it would work similarly at the Flight level.

[asfimport]

David Li / @lidavidm:
Ah, I misunderstood then, thanks for clarifying.

[asfimport]

Kazuaki Ishizaki / @kiszk:
@pitrou Thank you. In another issue, I am suspecting the endianness in schema. I will look at this.

[asfimport]

Kazuaki Ishizaki / @kiszk:
This fix is already created here. We confirmed this issue was solved by this fix.

The challenge is to create a test case. I realized that the latest arrow requires more native libraries for the build. It takes some time. So, I will create a test case with an older version arrow.

[asfimport]

Ravi Gummadi:
Thanks @kiszk  

[asfimport]

Bryan Cutler / @BryanCutler:
@kiszk since your patch looks to be pretty simple and low risk, what do you think about making a PR for it as is? If it passes current tests, we know it doesn't mess up anything for little endian. Then we could have a separate task to add a proper test with big endian machine, if that requires more effort?

[asfimport]

Kazuaki Ishizaki / @kiszk:
@BryanCutler Thank you for your suggestion. Sure, I will create a simple PR as is, and will open an follow-up issue to add a proper test.

[asfimport]

Bryan Cutler / @BryanCutler:
Issue resolved by pull request 12777
#12777