PERF: ArrowExtensionArray.__iter__

[lukemanley]

• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/v2.0.0.rst file if fixing a bug or adding a new feature.

Faster iteration through pyarrow-backed arrays.

from asv_bench.benchmarks.strings import Iter

dtype = "string[pyarrow]"
bench = Iter()
bench.setup(dtype)

%timeit bench.time_iter(dtype)

# 266 ms ± 6.66 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)    <- main
# 57.7 ms ± 679 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  <- PR

[mroeschke]

Is there any benefit to combining the chunks of self._data and then iterating?

[lukemanley]

Maybe seeing a 5-10% improvement testing locally, but not much. Seems to be 5-10% with 10 chunks or 10000 chunks. I can add it if you want?

[mroeschke]

Does the 5%-10% hold if there are a low number of chunks (1, 2, or 3)? I'd guess that most of the time there shouldn't be large amount of chunks since most construction is pd.chunked_array([scalars])

[lukemanley]

I don't think I'm seeing a consistent improvement at low number of chunks.

If curious, here is the snippet I'm testing with:

import pyarrow as pa

def time_iter(arr):
    for v in arr:
        pass

num_chunks = 2
chunk_size = 10**5
data = [np.arange(chunk_size)] * num_chunks
arr = pa.chunked_array(data)

%timeit time_iter(arr)
%timeit time_iter(arr.combine_chunks())

[jbrockmendel]

IIUC doing combine_chunks will cause a copy, so any speed improvements will be offset by memory increase. if so, i think not worth it

[mroeschke]

Thanks @lukemanley!