DOC: extension array with arrow

[achapkowski]

Pandas version checks

• I have checked that the issue still exists on the latest versions of the docs on main here

Location of the documentation

With the new 2.0 improvements, is there anything that needs to be added to custom dtypes and extension arrays to allow them to take advantage of the 2.x speed ups?

I couldn't find doc on this.

Documentation problem

Extension array documentation is light

Suggested fix for documentation

Answer the question above on how to integrate with the arrow backend.

[phofl]

Hi, https://pandas.pydata.org/docs/dev/user_guide/pyarrow.html?

[achapkowski]

That covers the standard dtypes not the custom ones.

It talks about how it's implemented but doesn't go into enough detail. Also is there a way to always use the arrow backend or do you have to always set the global settings?

[mroeschke]

is there anything that needs to be added to custom dtypes and extension arrays to allow them to take advantage of the 2.x speed ups?

Do you mean when using a pyarrow.ExtensionType or pandas.ExtensionDtype?

[jorisvandenbossche]

With the new 2.0 improvements, is there anything that needs to be added to custom dtypes and extension arrays to allow them to take advantage of the 2.x speed ups?

The arrow-based data types are ExtensionArrays themselves. So if you are implementing your own custom extension array, that won't automatically make any use of that (different EA implementations are independent, and you EA can't be two EAs at the same time). So if you want to make use of pyarrow under the hood in your own custom EA, you currently need to implement that yourself.

It's an interesting question how we can make this easier in the future, though. pd.arrays.ArrowExtensionArray is publicly exposed, so you could subclass that, but I am not sure if we would already recommend that right now (given that this is still quite experimental / moving fast).

[achapkowski]

@jorisvandenbossche thanks for that! I'll give it a whirl and report back just to check the feasibility.

[jbrockmendel]

Is there anything actionable here?

[mroeschke]

Closing as not sure if there's anything else needed, but feel free to open up a new issue if you find something

[achapkowski]

@mroeschke was the doc updated?