ARROW-7031: [Python] Expose the offsets of a ListArray in python

[jorisvandenbossche]

No description provided.

[jorisvandenbossche]

Should this rather be implemented in C++ (there is currently no method on ListArray in C++ that directly returns the offsets as an arrow::Array)

[pitrou]

Right, that would be better IMO.

[wesm]

Easy to do, picking a name for such a method maybe is the hardest part.

We have

ListArray::values -> shared_ptr<Array>
ListArray::value_offset -> int32_t
ListArray::value_offsets -> shared_ptr<Buffer>
ListArray::raw_value_offsets -> const int32_t*

Thoughts about what to call it?

[pitrou]

ListArray::offsets_array?

[jorisvandenbossche]

As I noted in the issue, the current ListArray.values (and now also the ListArray.offsets) does not take the Array.offset into account. Personally, I think it would be useful to have some method to get the values/offsets that do take that into account, so the python does not need to be aware of sliced arrays.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-7031

[kszucs]

@jorisvandenbossche I think it makes sense to follow the semantics of the current implemenation. Could you please create an issue from your JIRA comment to track it?

[wesm]

Hm, I think we should probably slices the offsets so that arr.offsets[0] is the offset of the first element in arr.values. They aren't strictly comparable because the offsets are relative to values.

So to summarize:

• Slice result of offsets
• Do not slice result of values

[jorisvandenbossche]

@wesm I implemented the behaviour you proposed (in Python for now, first making sure I understood correctly). So we now have:

In [35]: arr = pa.ListArray.from_arrays(offsets=[0, 3, 5], values=[1, 2, 3, 4, 5])  

In [36]: arr  
Out[36]: 
<pyarrow.lib.ListArray object at 0x7ff2e6406be8>
[
  [
    1,
    2,
    3
  ],
  [
    4,
    5
  ]
]

In [37]: arr.values  
Out[37]: 
<pyarrow.lib.Int64Array object at 0x7ff2d1d813a8>
[
  1,
  2,
  3,
  4,
  5
]

In [38]: arr.offsets 
Out[38]: 
<pyarrow.lib.Int32Array object at 0x7ff2e6406b28>
[
  0,
  3,
  5
]

In [39]: arr2 = arr[1:]   

In [40]: arr2  
Out[40]: 
<pyarrow.lib.ListArray object at 0x7ff2e6406c48>
[
  [
    4,
    5
  ]
]

# values still return the full buffer (same as for arr)
In [41]: arr2.values 
Out[41]: 
<pyarrow.lib.Int64Array object at 0x7ff2e6406b88>
[
  1,
  2,
  3,
  4,
  5
]

# but the offsets are sliced, so you can use this to correctly index 
# the values for this sliced array
In [42]: arr2.offsets 
Out[42]: 
<pyarrow.lib.Int32Array object at 0x7ff2d1d811c8>
[
  3,
  5
]

In [43]: arr2.values[3:5]  
Out[43]: 
<pyarrow.lib.Int64Array object at 0x7ff2e6406dc8>
[
  4,
  5
]

That looks correct?

[wesm]

Yes, that looks right to me.

+1. We can handle the C++ issue separately

[jorisvandenbossche]

OK, created https://issues.apache.org/jira/browse/ARROW-7068 for the C++ side