[Python] How to add one level of nesting to flat table?

[sergun]

Describe the usage question you have. Please include as many useful details as possible.

I have flat pa.Table:

table = pa.table({"a": [1, 2, 3], "b": [3, 4, 5]})

How can I create new table from this one by adding one level of nesting?
So I want to have a new table with only one column "c" of type struct with two fields "a" and "b" and keep data from original table.

Component(s)

Python

[AlenkaF]

If you can work with record batches I would suggest using to_struct_array() method:

import pyarrow as pa
batch = pa.RecordBatch.from_pydict({"a": [1, 2, 3], "b": [3, 4, 5]})
struct_array = batch.to_struct_array()
batch_result = pa.RecordBatch.from_arrays([struct_array], names=["c"])
# pyarrow.RecordBatch
# c: struct<a: int64, b: int64>
#   child 0, a: int64
#   child 1, b: int64
# ----
# c: -- is_valid: all not null
# -- child 0 type: int64
# [1,2,3]
# -- child 1 type: int64
# [3,4,5]

If you need to work with tables then you can do the same for each individual chunk:

# I think this should work
table = pa.table({"a": [1, 2, 3], "b": [3, 4, 5]})
batches = []
for b in table.to_batches():
    batches.append(pa.RecordBatch.from_arrays([b.to_struct_array()], names=["c"]))
table_result = pa.Table.from_batches(batches)
# pyarrow.Table
# c: struct<a: int64, b: int64>
#   child 0, a: int64
#   child 1, b: int64
# ----
# c: [
#   -- is_valid: all not null
#   -- child 0 type: int64
# [1,2,3]
#   -- child 1 type: int64
# [3,4,5]]

[sergun]

If you can work with record batches I would suggest using to_struct_array() method:

import pyarrow as pa
batch = pa.RecordBatch.from_pydict({"a": [1, 2, 3], "b": [3, 4, 5]})
struct_array = batch.to_struct_array()
batch_result = pa.RecordBatch.from_arrays([struct_array], names=["c"])
# pyarrow.RecordBatch
# c: struct<a: int64, b: int64>
#   child 0, a: int64
#   child 1, b: int64
# ----
# c: -- is_valid: all not null
# -- child 0 type: int64
# [1,2,3]
# -- child 1 type: int64
# [3,4,5]

If you need to work with tables then you can do the same for each individual chunk:

# I think this should work
table = pa.table({"a": [1, 2, 3], "b": [3, 4, 5]})
batches = []
for b in table.to_batches():
    batches.append(pa.RecordBatch.from_arrays([b.to_struct_array()], names=["c"]))
table_result = pa.Table.from_batches(batches)
# pyarrow.Table
# c: struct<a: int64, b: int64>
#   child 0, a: int64
#   child 1, b: int64
# ----
# c: [
#   -- is_valid: all not null
#   -- child 0 type: int64
# [1,2,3]
#   -- child 1 type: int64
# [3,4,5]]

Thanks a lot @AlenkaF !

Am I right such transformations Table <-> Batches cost close to zero according:
https://arrow.apache.org/docs/cpp/tables.html#record-batches
?

"However, a table can be converted to and built from a sequence of record batches easily without needing to copy the underlying array buffers. A table can be streamed as an arbitrary number of record batches using a arrow::TableBatchReader. Conversely, a logical sequence of record batches can be assembled to form a table using one of the arrow::Table::FromRecordBatches() factory function overloads."

[AlenkaF]

Table to/from RecordBatches transformations are zero-copy.

[jorisvandenbossche]

We might want to add to_struct_array to Table as well? (returning a ChunkedArray of struct type) To make this a bit more convenient in case of a table.

[AlenkaF]

Oh, forgot the issue already exists with an open PR! :)#38520