[Format] Tracking for ensuring flatbuffer serialized values are aligned in stream/files.

[asfimport]

Overall tracking bug for implementation for IPC/File format proposed by: https://github.com/apache/arrow/pull/4951/files

 

Implementations must support backwards compatibility with the old format (and ideally do memcopies when required to avoid undefined behavior).  Having a backwards compatible write mode is optional

Reporter: Micah Kornfield / @emkornfield
Assignee: Wes McKinney / @wesm

Subtasks:

• [C++] Implement changes to ensure flatbuffer alignment.
• [Java] Make change to ensure flatbuffer reads are aligned
• [Go] Make change to ensure flatbuffer reads are aligned
• [JS] Implement changes to ensure flatbuffer alignment
• [Integration] Update integration test to use generated binaries to ensure backwards compatibility
• [Python] Provide mechanism for python to write out old format
• [Java] Use IPC continuation token to mark EOS
• [R] Provide mechanism to write out old format

Related issues:

• [Format][C++] Use two-part EOS and amend Format documentation (is a parent of)
• [Go] Update Go IPC writer to use two-part EOS per mailing list discussion (is a parent of)
• [Rust] Implement changes to ensure flatbuffer alignment (incorporates)

PRs and other links:

• GitHub Pull Request #5420

_{Note: This issue was originally created as ARROW-6313. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
This was all merged to master