Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] Support Flight RPC among diffent endian platforms #24942

Closed
asfimport opened this issue May 14, 2020 · 5 comments
Closed

[C++] Support Flight RPC among diffent endian platforms #24942

asfimport opened this issue May 14, 2020 · 5 comments

Comments

@asfimport
Copy link
Collaborator

This PR is derived from discussion in ARROW-8790. The current implementation of the Flight RPC exchanges messages using a native endlian format. It does not allow us to exchange data though Flight RPC among different endian machines (e.g. x86_64 <-> s390x).

There is a field to show an endianness at https://github.com/apache/arrow/blob/master/format/Schema.fbs#L343-L346, which is added by ARROW-245.

Reporter: Kazuaki Ishizaki / @kiszk
Assignee: Kazuaki Ishizaki / @kiszk

PRs and other links:

Note: This issue was originally created as ARROW-8797. Please see the migration documentation for further details.

@asfimport
Copy link
Collaborator Author

Kazuaki Ishizaki / @kiszk:
@pitrou According to your comment, current FlightRPC transfers data in native endlian. In my opinion, we may have two opinions to exchange data between different endian platforms.

  1. Exchange data using native endian. If the receiver detects the message endian is different from the machine-native endian, the receiver converts endian.
  2. The FlightRPC always use the specific endian regardless of endian of sender or receiver.

Is there any comment?

@asfimport
Copy link
Collaborator Author

Antoine Pitrou / @pitrou:
This is a general issue with our IPC implementation. The IPC format specifies the data endianness in the Schema structure:
https://github.com/apache/arrow/blob/master/format/Schema.fbs#L343-L346

However, this is currently ignored in the implemenation, so it will have to be fixed.

@asfimport
Copy link
Collaborator Author

Kazuaki Ishizaki / @kiszk:
Sure, IPC should exchange data using the data endianness in the schema. Then, a reader has to take care of the field.

First, we may have to create test cases.

@asfimport
Copy link
Collaborator Author

Kazuaki Ishizaki / @kiszk:
#7507 is trying to create test cases.

@asfimport
Copy link
Collaborator Author

Antoine Pitrou / @pitrou:
Issue resolved by pull request 7507
#7507

@asfimport asfimport added this to the 4.0.0 milestone Jan 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants