[C++/Python] Pandas roundtrip for date objects

[asfimport]

Arrow currently casts date objects to nanosecond precision datetime objects. I'd like to have a way to preserve the type during a roundtrip

>>> import pandas as pd
>>> import pyarrow as pa
>>> import datetime
>>> pa.date32().to_pandas_dtype()
dtype('<M8[ns]')
>>> df = pd.DataFrame({'date': [datetime.date(2018, 1, 1)]})
>>> df.dtypes
date object
dtype: object
>>> df_rountrip = pa.Table.from_pandas(df).to_pandas()
>>> df_rountrip.dtypes
date    datetime64[ns]
dtype: object

I'd expect something like this to work:

>>> import pandas.testing as pdt
>>> df_rountrip = pa.Table.from_pandas(df).to_pandas(date_as_object=True)
>>> pdt.assert_frame_equal(df_rountrip, df)

Reporter: Florian Jetter / @fjetter
Assignee: Krisztian Szucs / @kszucs

Related issues:

• pyarrow table from/to dataframe is type-lossy (is duplicated by)

PRs and other links:

• GitHub Pull Request #2535

_{Note: This issue was originally created as ARROW-2646. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
@fjetter would you like to do this for 0.10.0? We might consider some kind of Options object (like we have in C++) to manage all of these conversion parameters that may show up in many functions (to avoid having to maintain a lot of the same function signatures)

[asfimport]

Uwe Korn / @xhochy:
@wesm I'm still not convinced that an Options object would work well with Python users. Do you have an example of a Python project that uses it? I would expect that users would much more favour kwargs are done in Pandas/Matplotlib/… . While there are certain drawbacks for a library developer, I think it will be much more appreciated by users.

[asfimport]

Antoine Pitrou / @pitrou:
The standard csv module has both a notion of "dialect" and additional **kwargs to each function so that you can override individual options. Intuitively, it allows accepting individual option arguments without listing and documenting them explicitly for each method.

I tend to prefer the options object / dialect approach myself, but it's true I'm more in the library developer camp :-)

[asfimport]

Krisztian Szucs / @kszucs:
I do prefer kwargs on python side. We could introduce mixins to reduce the redundant function signatures.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 2535
#2535