Assigning Accessor to a specific dtype?

[achapkowski]

Problem description

With the v.23 accessors, I think they are really great, but for Series accessors, can you limit the functionality to just a specific dtype or a custom dtype?

Take the example on the page, the geo accessor.

Currently this applies to the whole series (if a series accessor is used), but if the functionality is limited to operate on a specific data type, why should the accessor be accessible to say an int or string type series.

I am trying to move away from extending by inheritance, but I want to limit the scope of the operations.

So going back to the geo sample, say I create a dtype extension called SpatialType or something like that, and then assign it to a column. I would like then register that functionality to that type only.

so:

@pd.api.extensions.register_series_accessor("geo", dtype="SpatialExt")
class GeoAccessor(object):
    def __init__(self, pandas_obj):
        self._obj = pandas_obj

    @property
    def center(self):
        # return the geographic center point of this DataFrame
        lat = self._obj.latitude
        lon = self._obj.longitude
        return (float(lon.mean()), float(lat.mean()))

    def plot(self):
        # plot this array's data on a map, e.g., using Cartopy
        pass

data = [
   [1,2,113,{'x':-75, 'y':35}],
   [2,20,213,{'x':-76, 'y':36}],
   [3,-20,123,{'x':-77, 'y':37}]
]
columns = ['OID', 'Elevation', 'measurement', 'geometry']
df = pd.DataFrame(data, columns)
df.set_geometry('geometry') # define the spatial column using some dataframe accessor

df.geometry.geo.plot() # This would show
df.OID.geo.plot() # this would not work.

This would provide a more controlled extension platform allowing users to limit the extensions to defined types.

[TomAugspurger]

What kind of behavior do you want to achieve? For most things, raises an AttributeError in the accessors __init__ achieves the same behavior as the accessor not being there at all. That's what we do for .str:

In [8]: 'str' in dir(pd.Series([1, 2]))
Out[8]: False

In [9]: 'str' in dir(pd.Series(['a', 'b']))
Out[9]: True

In [10]: pd.Series([1, 2]).str

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-10-e5754c481fcd> in <module>()
----> 1 pd.Series([1, 2]).str

~/sandbox/pandas-ip/pandas/pandas/core/generic.py in __getattr__(self, name)
   4372         if (name in self._internal_names_set or name in self._metadata or
   4373                 name in self._accessors):
-> 4374             return object.__getattribute__(self, name)
   4375         else:
   4376             if name in self._info_axis:

~/sandbox/pandas-ip/pandas/pandas/core/accessor.py in __get__(self, obj, cls)
    130             # we're accessing the attribute of the class, i.e., Dataset.geo
    131             return self._accessor
--> 132         accessor_obj = self._accessor(obj)
    133         # Replace the property with the accessor object. Inspired by:
    134         # http://www.pydanny.com/cached-property.html

~/sandbox/pandas-ip/pandas/pandas/core/strings.py in __init__(self, data)
   1775
   1776     def __init__(self, data):
-> 1777         self._validate(data)
   1778         self._is_categorical = is_categorical_dtype(data)
   1779         self._data = data.cat.categories if self._is_categorical else data

~/sandbox/pandas-ip/pandas/pandas/core/strings.py in _validate(data)
   1795             # (instead of test for object dtype), but that isn't practical for
   1796             # performance reasons until we have a str dtype (GH 9343)
-> 1797             raise AttributeError("Can only use .str accessor with string "
   1798                                  "values, which use np.object_ dtype in "
   1799                                  "pandas")

AttributeError: Can only use .str accessor with string values, which use np.object_ dtype in pandas

[TomAugspurger]

We document this expectation in the API docs: http://pandas-docs.github.io/pandas-docs-travis/generated/pandas.api.extensions.register_dataframe_accessor.html though it isn't mentioned here: http://pandas-docs.github.io/pandas-docs-travis/extending.html. It'd be good to add a note saying that developers / users should expect an AttributeError to be raised if the types don't work out.

[achapkowski]

@TomAugspurger Thanks for the quick response. I was looking at your cyberpandas project. The project seems to do what I'm looking for but I was trying to figure out what is required and what was not. Also, when you should you put functionality on the dtype vs on the accessor?

@staticmethod
    def _validate(obj):
        if not is_ipaddress_type(obj):
            raise AttributeError("Cannot use 'ip' accessor on objects of "
                                 "dtype '{}'.".format(obj.dtype))

[TomAugspurger]

I was trying to figure out what is required and what was not.

I don't think anything is really required. The accessor's __init__ method get's the DataFrame / Series / Index, and is free to do whatever it wants at that point. It's really about doing the expected (i.e. what pandas does with its own accessors).

when you should you put functionality on the dtype vs on the accessor?

Can you clarify that question? I don't quite follow. As far as this issue is concerned, the only important thing is that your accessor's __init__ method does whatever type checking it needs. For whatever reason, in cyberpandas's accessor I broke that out to a separate method (maybe it's called from somewhere else). But it's called from the __init__ so things should work as expected.

[achapkowski]

@TomAugspurger ok let me clarify.

So if I make a geo on a series accessor. Should I put all geo operations on the accessor or should I put the geo operations on the custom dtype?

[TomAugspurger]

Ah, I see. I'll use "extension array" instead of "custom dtype", since we have an ExtensionDtype class as well :)

Accessors can be used with regular dtypes as well, not just extension ones. This is especially relevant for DataFrames, which won't have a single array type backing them. That's a caveat to my next recommendation.

For cyberpandas, I've only registered a Series accessor, and I've put all the complicated logic on the extension array itself. I think this has worked well. IMO, the accessor should be responsible only for

1. providing the user an API for custom methods
2. unboxing series / frame / index to your custom array
3. boxing the result in a series / frame / index

This means the logic in https://github.com/ContinuumIO/cyberpandas/blob/master/cyberpandas/_accessor.py can be re-used for multiple accessors, since it's just being used for boxing / unboxing, nothing to do with IP addresses specifically.

[achapkowski]

ok great, i get it. 👍

[TomAugspurger]

@achapkowski are you interested in updating the docs in http://pandas-docs.github.io/pandas-docs-travis/extending.html to say that the accessor should do type validation in its __init__, raising an AttributeError if it isn't valid?

[achapkowski]

@TomAugspurger Sure I'll add this code.