[Data] Avoid per-column Series materialization in tensor-column casting

[ayushk7102]

Description

When we are converting tensor columns of a block/batch to ndarrays and vice versa, our current implementation scans over the columns using df.items() and checks the dtype of each column. This is wasteful as we pay the cost of materializing the column as a pd.Series just to check its dtype. Instead, what we can do is iterate over the df.dtypes.items() which only yields numpy dtype objects.
_cast_tensor_columns_to_ndarrays and _cast_ndarray_columns_to_tensor_extension in data_batch_conversion.py .items(). df.items() materializes a pandas Series for every column just so the body can read its dtype: which is wasteful when most (or all) columns aren't tensor columns

Additional Information

Code paths

function	hot path	frequency
_cast_tensor_columns_to_ndarrays	ArrowBlockAccessor.to_pandas()	per batch in iter_batches(pandas)
_cast_ndarray_columns_to_tensor_extension	TableBlockBuilder.build() → _combine_tables()	per block built

TODO: Add the microbenchmark

[gemini-code-assist]

Code Review

This pull request optimizes pandas DataFrame conversions by scanning df.dtypes instead of df.items(), avoiding the unnecessary materialization of Series objects for non-tensor columns. The reviewer identified a critical issue where duplicate column names in the DataFrame would cause df[col_name] to return a DataFrame instead of a Series, leading to runtime errors. To resolve this, the reviewer suggested iterating with enumerate(df.dtypes.items()) and retrieving columns safely by index using df.iloc[:, i].

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[ayushk7102]

@goutamvenkat-anyscale A few comments to address conversion overhead:

Note that we cannot df[col_name].iloc[0] in the _cast_ndarray_columns_to_tensor_extension() path because numpy columns all report their dtypes as numpy.object_. However, N-D arrays all will report type(s.iloc[0]) as numpy.ndarray which is our only source of truth that this should be backed by a TensorDtype, rather than as a list of strings, scalars, etc. Also, df[col_name].iloc[0] should return a view over the first element in the buffer, which is cheaper than actually materializing the entire series.

This is different from the cast_tensor_columns_to_ndarrays() path where we already have the columns as TensorArrays, such that the check becomes a cheap isinstance(TensorDtype).

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit b71c050. Configure here.}