BUG: Inconsistent datetime conversion behavior when constructing a DataFrame with Python datetimes.

[raeganbarker]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

from datetime import datetime

from pandas import DataFrame

date = datetime.now()

data = {
    'date1': date,
    'date2': [date, date],
}

df = DataFrame(data)

print(df.dtypes)

Issue Description

The date columns are not the same type (The case in 2.1.0).

date1    datetime64[us]
date2    datetime64[ns]

Expected Behavior

The date columns are the same type (The case in 2.0.3).

date1    datetime64[ns]
date2    datetime64[ns]

Installed Versions

INSTALLED VERSIONS ------------------ commit : ba1cccd python : 3.11.4.final.0 python-bits : 64 OS : Linux OS-release : 5.15.49-linuxkit-pr Version : #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 machine : aarch64 processor : byteorder : little LC_ALL : None LANG : C.UTF-8 LOCALE : en_US.UTF-8

pandas : 2.1.0
numpy : 1.25.2
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 68.0.0
pip : 23.2.1
Cython : None
pytest : 7.4.0
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.1.2
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 2.0.20
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[carlosProgrammer]

take

[IzerOnadimQC]

I think a conversation is needed regarding the expected behaviour in Pandas 2 when instantiating a DataFrame with columns of type dt.datetime. In Pandas<2, it is clear that dt.datetime types are expected to be inferred as datetime64[ns] - but it is unclear whether this expectation still holds for Pandas>=2.

As far as I can tell, the discrepancy was caused by this PR: #52212, which changes the way dt.datetime types are inferred. Previously, they were used to instantiate a Timestamp which was then cast to ns units, but now, there is no cast, meaning the Timestamp resolution will remain as us, which appears to be the default resolution when a Timestamp is created using a dt.datetime. I don't think any corresponding change has been made when the DataFrame column is constructed using a list, hence the discrepancy.

@jbrockmendel since you authored that change I was wondering if you had any thoughts on what type should be inferred from dt.datetime in Pandas>=2? The bug (#51196) that the PR addressed was regarding Timestamp types with non-nano units being inferred as datetime64[ns], hence, is it possible that the change to how dt.datetime types are handled was a mistake? If so, I'd be happy to make a fix where dt.datetime types are inferred with nanosecond resolution by default, separate from how Timestamps or other types that have their units explicitly set to non-nano resolutions are handled. @MarcoGorelli it would also be interesting to get your opinion on this matter since you reported the issue to begin with.

[MarcoGorelli]

hey - I think the current "rule" is that:

• for scalars, the resolution is preserved (so for stdlib datetime, it becomes 'us', because that's the resolution of the python stdlib)
• for a list, the resolution is 'ns' by default

[IzerOnadimQC]

• for scalars, the resolution is preserved (so for stdlib datetime, it becomes 'us', because that's the resolution of the python stdlib)
• for a list, the resolution is 'ns' by default

Does this mean that the above discrepancy is actually not considered a bug? I.e. it is expected behaviour for the two columns in the DataFrame to have different dtypes?

[MarcoGorelli]

I think @jbrockmendel was suggesting to auto-infer the resolution in the list case as well, but that the implementation is quite tricky

[jbrockmendel]

Marco has it right. The ideal solution is to improve array_to_datetime/maybe_convert_objects to do resolution inference. This would cause the example in the OP to have datetime64[us] dtype for both columns.

Actually implementing this is difficult (particularly avoiding major performance regressions), but increasingly high on my todo list. I'm optimistic it will be ready for 2.2.