How to Use Modin with ExtensionArrays and Accessors?

[achapkowski]

Pandas has an extension framework, how do you register the accessors and extension arrays with modin? My API has lots of custom dtypes and accessors to do specialized analysis and it would be great if modin honored those.

see: https://pandas.pydata.org/docs/development/extending.html

[YarShev]

Hi @achapkowski, thanks for the question! We merged PR #6961 on this matter last week. You can use register_dataframe_accessor, register_series_accessor or register_pd_accessor to register a custom accessor for a DataFrame, Series object or pd module, respectively. Just an example from our tests.

import modin.pandas as pd
from modin.pandas.api.extensions import register_dataframe_accessor

expected_string_val = "Some string value"
method_name = "new_method"
df = pd.DataFrame([1, 2, 3])

@register_dataframe_accessor(method_name)
def my_method_implementation(self):
    return expected_string_val

assert method_name in pd.dataframe._DATAFRAME_EXTENSIONS_.keys()
assert pd.dataframe._DATAFRAME_EXTENSIONS_[method_name] is my_method_implementation
assert df.new_method() == expected_string_val

Note that the feature is currently available in master only but we expect to release Modin 0.28.0 this week.

[achapkowski]

@YarShev this is great! thank you, I will test this once it is officially released.

[YarShev]

@achapkowski, sounds good! Feel free to reopen this issue or open a new one if you encounter any problem with the feature.