[Format][C++] Add "LargeList" type with 64-bit offsets

[asfimport]

Mentioned in #3845

Reporter: Wes McKinney / @wesm
Assignee: Antoine Pitrou / @pitrou

Related issues:

• [Python] Support LargeList (blocks)
• [C++] Implement casting List <-> LargeList (blocks)
• [Format] Add LargeBinary and LargeString types (relates to)

PRs and other links:

• GitHub Pull Request #3848
• GitHub Pull Request #4969

_{Note: This issue was originally created as ARROW-4810. Please see the migration documentation for further details.}

[asfimport]

Maarten Breddels / @maartenbreddels:

Having arrays with > 2GB elements or binary arrays with > 2GB of data would be considered an anti-pattern in the context of database systems, regardless of whether the offsets are 32- or 64-bit. So in light of these it doesn't make sense to have the default type be 64-bit capable if this capability is seldom used

I agree it's not the best idea, but people will find a reason to do it, and since there will not be a straightforward workaround, it may spin off another 'standard' :)

But, since allow/supporting it will solve both issues (>2GB elements, and less code complexity) I thought I would mention that as well.

 

As far as the implementation, are you thinking about a new class (apart from ArrayList), or does it seem feasible to include a type for the value_offsets?

[asfimport]

Wes McKinney / @wesm:
Yes, we'd want a separate container type, e.g. LargeListArray

[asfimport]

Maarten Breddels / @maartenbreddels:
I see BinaryArray/StringArray classes have similar implementation, the same holds for that, create a Large(Binary/String)Array?

[asfimport]

Wes McKinney / @wesm:
Yes – there's a fair bit of boilerplate to consider. If we don't care about IPC roundtrips for the time being then changes to the metadata specification can be addressed separately (https://github.com/apache/arrow/blob/master/format/Schema.fbs)

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
I'm not a afan of multiplying types, have we studied adding parameter to the existing type?

[asfimport]

Jacques Nadeau / @jacques-n:
I'm -1 on this (and any format change) unless there are first class implementations in both Java and C++. I think it is important to avoid having features that are only in one our reference implementations.

[asfimport]

Wes McKinney / @wesm:
Yes, I agree we should create an implementation for Java as well. Having to support 64-bit offsets is inevitable, but we should not accumulate technical debt by failing to create multiple reference implementations

[asfimport]

Philipp Moritz / @pcmoritz:
I agree that we should have a reference implementation in both C++ and Java. Is there anybody who can help with this? I don't personally have the expertise to do it unfortunately (and would also like to avoid this becoming a zombie project like https://issues.apache.org/jira/browse/ARROW-1692).

[asfimport]

Micah Kornfield / @emkornfield:
How does this reconcile with other 32 bit length limitations in the standard. I hacked a quick and dirty proof of concept in Java, but ran into the issue of a lot of APIs assuming int instead on long?

[asfimport]

Micah Kornfield / @emkornfield:
I see Message.fbs got updated 2-years ago to use 64-bit bit ints throughout.  But the layout still lists 32 bit ints.  This resolves my question I think.

[asfimport]

Micah Kornfield / @emkornfield:
Issue resolved by pull request 4969
#4969