BUG: Datetime subclasses are no longer datetime64 types

[fjetter]

Code Sample, a copy-pastable example if possible

>>> import pandas as pd
>>> from datetime import datetime
>>> class MyDatetime(datetime):
...             pass
>>> df_subclass = pd.DataFrame({"datetime": [MyDatetime(2018, 1, 1, 1, 1)]})
>>> df_subclass.dtypes
datetime    object
dtype: object

>>> df_dt = pd.DataFrame({"datetime": [datetime(2018, 1, 1, 1, 1)]})
>>> df_dt.dtypes
datetime    datetime64[ns]
dtype: object

Problem description

Since version 0.23.0 pandas treats subclasses of datetime no longer as datetime64[ns]

Expected Output

Same behavior as in pre 0.23.0 versions, i.e. subclasses of datetime are of type datetime64[ns]

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.0
pytest: 3.4.0
pip: 9.0.1
setuptools: 38.5.1
Cython: 0.27.3
numpy: 1.14.3
scipy: None
pyarrow: 0.8.1.dev163+gb33dfd9
xarray: None
IPython: 6.2.1
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[chris-b1]

@TomAugspurger, @jbrockmendel - I'm not sure this coercion of datetime subclasses was ever explicitly supported, but assuming something in the EA/tslib refactoring hit this?

[jbrockmendel]

@chris-b1 I'm pretty sure this is driven by a change in tslib.array_to_datetime that effectively assumes that whenever isinstance(val, datetime) and not (val.__class__ is datetime) then val is a pd.Timestamp object. As a result, when it sees a MyDatetime object it ends up looking up val.nanosecond, raises AttributeError, and falls back to an object-dtype.

Pinning nanosecond = 0 to MyDatetime "fixes" this for me:

import pandas as pd
from datetime import datetime

class MyDatetime(datetime):
    nanosecond = 0
    
df = pd.DataFrame({"datetime": [MyDatetime(2018, 1, 1, 1, 1)]})
>>> df.dtypes
datetime    datetime64[ns]
dtype: object

We can probably edit array_to_datetime to do a getattr(val, 'nanosecond', 0) lookup to avoid this issue.

[jreback]

agree with @jbrockmendel here, though note that forced conversion defeats the purpose of a datetime subclass, if you actually need one (and not really sure why, can you elaborate?) then you would be better off with an ExtensionArray.

[fjetter]

I ran into this issue during unit testing where I do not necessarily use the proper datetime object but rather a fake of some sorts. ExtensionArray would not be a good fit for this I suppose.

For instance, I am using https://github.com/spulec/freezegun which creates a FakeDatetime object where the proper datetime class is registered as a metaclass.

[mroeschke]

This looks to be fixed on master. Could use a test

In [68]: >>> import pandas as pd
    ...: >>> from datetime import datetime
    ...: >>> class MyDatetime(datetime):
    ...: ...             pass
    ...: >>> df_subclass = pd.DataFrame({"datetime": [MyDatetime(2018, 1, 1, 1, 1)]})
    ...: >>> df_subclass.dtypes
Out[68]:
datetime    datetime64[ns]
dtype: object

In [70]: pd.__version__
Out[70]: '1.1.0.dev0+1068.g49bc8d8c9'

[avinashpancham]

take

[avinashpancham]

Added test to verify the functionality for Datetime subclasses, but it still does not work for Date subclasses on master (see below). I don't have enough experience to figure out what needs to be changed to make this work. Can someone maybe point me in the right direction, then I can determine whether is it possible for me to do this.

In [1]: import pandas as pd                                                                       

In [2]: from datetime import date                                                                 

In [3]: class DateSubclass(date): 
   ...:     pass 
   ...:                                                                                           

In [4]: data = pd.DataFrame({"date": [DateSubclass(2020, 1, 1)]})                                 

In [5]: data.dtypes                                                                               
Out[5]: 
date    object
dtype: object