Skip to content

[Python] Convert maps to dicts without per-element Scalars in to_pylist(maps_as_pydicts=...) #50429

Description

@viirya

Describe the enhancement requested

Follow-up of #50326. GH-50327 makes Array.to_pylist() convert without per-element Scalars, but the maps_as_pydicts option still routes to the Scalar-based path: every map row allocates a MapScalar, converts its keys via per-element as_py, and builds the dict in Python.

Converting maps to dicts is the natural consumption pattern for engines whose map values are Python dicts (e.g. Spark's Arrow-serialized Python UDFs currently receive association lists from to_pylist() and rebuild a dict per row in pure Python — one of the dominant remaining costs in that path).

Proposal: thread maps_as_pydicts through the scalar-free _getitem_py mechanism introduced in GH-50327 so MapArray builds the dict directly from the flattened keys/items children:

  • default (None) semantics unchanged (association lists);
  • 'lossy'/'strict' build the dict in one pass; when the dict size shows duplicate keys, the row is redone with the careful per-key loop so the warning ('lossy') and KeyError ('strict') semantics stay exactly identical to MapScalar.as_py;
  • invalid values still raise the same ValueError when a map value is converted;
  • the option propagates through nested types (list/struct/map values) as before, and unspecialized types keep the exact Scalar fallback.

Component(s)

Python

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions