[Python] Table.to_pandas converts Arrow date32[day] to pandas datetime64[ns]

[asfimport]

This issue was raised here:

wesm/feather#359

I explored this minimally against Arrow master:

https://gist.github.com/wesm/2ebe0ca2461d1ecfba6185777238ad1f

While it's pretty memory-wasteful, it might be better to preserve the intent of the data type when converting to pandas data structures. It also allows the data to round trip successfully

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

Related issues:

• [Python] Set date_as_object to True in *.to_pandas as default after deduplicating logic implemented (relates to)
• [Python] Deduplicate datetime.date objects in Table.to_pandas internals (is related to)

_{Note: This issue was originally created as ARROW-3899. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
We can minimize memory use by passing the date32[day] values through a hash table. This should be easier with the new hashing machinery

[asfimport]

Uwe Korn / @xhochy:
We currently have the parameter date_as_object on the conversion to Pandas. This is set to false. Although I would like to have true as the default, this would be a heavy breaking change. We should add a DeprecationWarning that we will change that in the next release and then do it a release later.

[asfimport]

Wes McKinney / @wesm:
Oh right. Sounds reasonable. This is a bit tedious as we'll only want to warn when there is actually a date field

[asfimport]

Wes McKinney / @wesm:
This was resolved in 0.12. On master now

In [4]: arr = pa.array([date(2001, 1, 1), None, date(2001, 1, 2)])                                                                                                                             

In [5]: arr                                                                                                                                                                                    
Out[5]: 
<pyarrow.lib.Date32Array object at 0x7f033ceb7ae8>
[
  11323,
  null,
  11324
]

In [6]: arr.to_pandas()                                                                                                                                                                        
Out[6]: 
array([datetime.date(2001, 1, 1), None, datetime.date(2001, 1, 2)],
      dtype=object)