ArrayData buffers are inconsistent accross implementations

[asfimport]

ArrayData implementations seems to share close structure fields accross languages, but their usage is not consistent accross implementation.

 

Example using ListArray's offsets buffer, in C++, Rust and JavaScript implementation:

 - C++: offset's buffer is the second buffer (validity bitmap is first buffer, and buffers are laid in a type-dependant way) https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/array_nested.cc#L189

 - Rust: offset's buffer is the first buffer (validity bitmap is not part of the collection, and buffers are laid in a type-dependant way) https://github.com/apache/arrow/blob/master/rust/arrow/src/array/array_list.rs#L235

 - JavaScript: offset's buffer is the first buffer (they have fixed position)

arrow/js/src/data.ts

Line 125 in 8e43f23

(buffer = (buffers as Buffers<T>)[0]) && (this.valueOffsets = buffer);

 

Note that we have the same inconsistency for validity and data buffers.

 

This is important in my project because I would like to transport buffers list accross technologies, and ArrayData seemed the easiest structure to transport.

Reporter: Vincent Trumpff

_{Note: This issue was originally created as ARROW-12223. Please see the migration documentation for further details.}

[asfimport]

Andrew Lamb / @alamb:
Migrated to github: apache/arrow-rs#207

[asfimport]

Vincent Trumpff:
Thanks for pushing on it!

Note that there is not only inconsistency between rust implementation and the others, but also between C++ and JavaScript implementation.

[asfimport]

Vincent Trumpff:
Reopening as we still have difference between C++ and JavaScript unless I missed something?

 

Edit:

If the community do agree with the point, I would be happy to propose a PR on C++ and JavaScript sources.

[asfimport]

Micah Kornfield / @emkornfield:
As far as I know it has not been the goal of the project to have identical implementations of items not specified in the specification (IPC/File and FFI ABI).

[asfimport]

Antoine Pitrou / @pitrou:
Late answer, but I don't think this is a bug indeed. If you want to transport data between runtimes, there are two official solutions:

• the Arrow IPC protocol
• the Arrow C data interface (only for in-process data sharing, guaranteed zero-copy)