Reading column_data from RecordBatch in Cython

[darklord1807]

Hello,

I am trying tor read the column data from the CRecordBatch object, but it says Object of type 'CRecordBatch' has no attribute 'column_data'.

Example.pyx

from pyarrow.lib cimport *
from pyarrow cimport import_pyarrow
import_pyarrow()

def get_array_length(obj):
    cdef shared_ptr[CRecordBatch] record = pyarrow_unwrap_batch(obj)
    if record.get() == NULL:
        raise TypeError("not a batch")
    return record.get().column_data(1)

test.py

batch = pa.RecordBatch.from_pandas(df)
print(example.get_array_length(batch))

Full Error:

  tree = Parsing.p_module(s, pxd, full_module_name)

Error compiling Cython file:
------------------------------------------------------------
...
    cdef shared_ptr[CRecordBatch] record = pyarrow_unwrap_batch(obj)

    if record.get() == NULL:
        raise TypeError("not a batch")

    return record.get().column_data(1)
                      ^
------------------------------------------------------------

example.pyx:15:23: Object of type 'CRecordBatch' has no attribute 'column_data'
Traceback (most recent call last):
  File "setup.py", line 9, in <module>
    ext_modules = cythonize("example.pyx")
  File "/usr/local/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1102, in cythonize
    cythonize_one(*args)
  File "/usr/local/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1225, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: example.pyx

[edponce]

Hi @darklord1807, thanks for informing there is a missing method in RecordBatch. I created ARROW-14449 to track this issue.

[edponce]

You still can access the data (ArrayData) as follows:

def get_array_length(obj):
    cdef:
        shared_ptr[CRecordBatch] record = pyarrow_unwrap_batch(obj)
        shared_ptr[CArray] array
        shared_ptr[CArrayData] data
    if record.get() == NULL:
        raise TypeError("not a batch")
    array = record.get().column(1)
    data = array.get().data()
    return data.get().length

[emkornfield]

Closing as there is now a tracking issue and workaround posted.