Make Column.to_pandas return Index instead of Series

[mroeschke]

Description

Column.to_pandas backs Index.to_pandas/Series.to_pandas/DataFrame.to_pandas and returned a pandas.Series; however, the index of this pandas.Series was not strictly necessary for Index.to_pandas and DataFrame.to_pandas.

Additionally, pandas.Index is 1D-like like Column and provides a better mental model to to_pandas conversion.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[vyasr]

Does it really make sense for column objects to convert to pandas objects at all? I think we still need some sort of method at the column level so that columns of different dtypes can overload it to produce different results (e.g. CategoricalColumn needs to convert the categories as well as the labels), but is there a lower-level representation that we could change these methods to return instead so that we could delegate the actual pandas object construction to the Frame level? I'm imagining, in cudf.Series for example, something like return pd.Series(self._column.to_pandas_data()) instead of return self._column.to_pandas(). Does that make sense?

[mroeschke]

is there a lower-level representation that we could change these methods to return instead so that we could delegate the actual pandas object construction to the Frame level?

I'm fairly skeptical that there is a common lower-level representation. Generally Column.to_arrow is called first, then each column dtype manipulates that as needed (from just calling to_pandas to needing a Python list of objects).

I agree it would be nice to have the pandas construction happen at the Frame level, but I think Frame would have to inherit the column dtype specific logic depending on what self._column.to_pandas_data() returns which isn't great.

[vyasr]

Fair enough. I'd file this in the back of your head in the list of "things we should consider when rewriting cudf internals to use pylibcudf". I suspect that we'll want to think hard about problems like this when reenvisioning what the column layer of cudf ought to look like.

[mroeschke]

Thanks. Yeah I think for this use case converting a pylibcudf column to something arrow-like is probably the correct "common data" for the to_pandas API in the future, especially if pandas gains pyarrow as a required dependency

[mroeschke]

/merge