[Python/Question] Maximum length of an Array created from ndarray

[asfimport]

So the format describes that an array max length is 2^31 - 1, however the following python snippet creates a 2**32 length arrow array:

a = np.ones((2**32,), dtype='int8')
A = pa.Array.from_pandas(a)
type(A)

pyarrow.lib.Int8Array

Based the layout specification I'd expect a ChunkedArray of three Int8Array's with lengths:
[2^31 - 1, 2^31 - 1, 2] or should raise an exception?

If it's the expectation is there any documentation for it?

Reporter: Krisztian Szucs / @kszucs

_{Note: This issue was originally created as ARROW-2476. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
I guess the spec is incorrect or obsolete, since at least in the C++ (and Python) implementation arrays a 64-bit length (signed, but it doesn't matter much).

The 32-bit limitation does apply to the underlying data of list arrays (and therefore also string and binary arrays).

[asfimport]

Krisztian Szucs / @kszucs:
In conjunction with https://issues.apache.org/jira/browse/ARROW-2367 now I'm totally confused.

Are the array lengths arrow format specific or implementation specific?
What are the actual limits of an array? Why are these limits datatype dependent?

The list offset bulletpoint in the format is also misleading.

[asfimport]

Uwe Korn / @xhochy:
I think the array size restriction is in place as other implementation like Java are unable to support.

[asfimport]

Antoine Pitrou / @pitrou:

Are the array lengths arrow format specific or implementation specific?

I don't know. But array length is not part of array data, only metadata, so I guess implementations are free to represent it as they prefer?

The list offset bulletpoint in the format is also misleading.

Can you explain what looks misleading to you? 

[asfimport]

Krisztian Szucs / @kszucs:
Quoting from the spec:

"Any array has a known and fixed length, stored as a 32-bit signed integer, so a maximum of 2^31 - 1 elements."
Conclusion:

max_len(array) == 2**31 - 1

"An offsets buffer containing 32-bit signed integers with length equal to the length of the top-level array plus one. Note that this limits the size of the values array to 2^31-1"
Conclusion:

len(offsets) == len(array) + 1
max_len(array) == 2**31 - 1 

# => thus
max_len(offsets) == 2**31  

In builder.h

constexpr int64_t kBinaryMemoryLimit = std::numeric_limits<int32_t>::max() - 1;
constexpr int64_t kListMaximumElements = std::numeric_limits<int32_t>::max() - 1;
// where both are actually (2**31 - 1) - 1

And according to Antoine's first comment cpp and python impl uses (2^63 - 1) instead of (2^31 - 1) in case of primitive types but not for list, string, binary.
Why is that limitation for list-like arrays but not for primitive ones?

As a sidenote the length of the offsets buffer can't be represented in a 32-bit signed integer (for an array with maximum (2^31 - 1) number of element), it requires a 64 bit signed integer like in the implementation. This also means that theoretically a list array with (2^63 - 1) element requires a buffer with maximum 2^63 number of elements, representable as int128 (of course we won't hit that limit).

I mean the values are correct (plus/minus one), but the limits (or the limitlessness) should be documented somewhere - clearly and consistently.

[asfimport]

Antoine Pitrou / @pitrou:
The offsets array is part of the data and needs to be interchangeable, so an implementation cannot use an array of int64_t if the spec says it's an array of int32_t.

As a sidenote the length of the offsets buffer can't be represented in a 32-bit signed integer (for an array with maximum (2^31 - 1) number of element), it requires a 64 bit signed integer
You seem to be confusing length in elements and size in bytes here. The spec says the number of elements is limited to 2**31, not the size in bytes.

(the 32-bit limitation doesn't look like a good idea to me, but I may be missing some context)

[asfimport]

Krisztian Szucs / @kszucs:
I wasn't thinking of the datatype the array holds, just the maximum length of the arrays.
In the sidenote ment the number of offsets (just can't find the right impl line).

According to array lengths section I'd think that each array can have at most (2^31 - 1) elements.

Why does it apply to the List<List> example (including the top array and the nested ones) but doesn't to an array of primitive type like array<int32>?

[asfimport]

Wes McKinney / @wesm:
This was clarified in the format documentation