BUG: AttributeError: 'DatetimeArray' object has no attribute 'notna' in pandas 2.1.1

[davidboweninrupt]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import datetime

pd.DataFrame( [datetime.datetime.now(datetime.UTC)])[0].value_counts(bins=2)

Issue Description

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib64/python3.11/site-packages/pandas/core/base.py", line 1010, in value_counts
    return algorithms.value_counts_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/pandas/core/algorithms.py", line 885, in value_counts_internal
    ii = cut(values, bins, include_lowest=True)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/pandas/core/reshape/tile.py", line 247, in cut
    x, dtype = _coerce_to_type(x)
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.11/site-packages/pandas/core/reshape/tile.py", line 505, in _coerce_to_type
    x = np.where(x.notna(), x.view(np.int64), np.nan)
                 ^^^^^^^
AttributeError: 'DatetimeArray' object has no attribute 'notna'

Note that a potential workaround is to convert the datetime to int and then handle the fallout of that in subsequent code.

pd.DataFrame( [datetime.datetime.now(datetime.UTC)])[0].apply(datetime.datetime.timestamp).value_counts(bins=2)
(1694934883.4420002, 1696631514.958]    1
(1696631514.958, 1698328146.473]        0
Name: count, dtype: int64

Expected Behavior

In pandas 2.1.0 we used to get the binned counts back, such as:

self = 0
(2023-09-17 07:09:20.929712127, 2023-10-06 22:26:32.121833984]    1
(2023-10-06 22:26:32.121833984, 2023-10-26 13:43:43.313955840]    0
Name: count, dtype: int64

Installed Versions

pd.show_versions()
/usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS
------------------
commit              : e86ed377639948c64c429059127bcf5b359ab6be
python              : 3.11.5.final.0
python-bits         : 64
OS                  : Linux
OS-release          : 6.5.5-200.bz2227221.fc38.x86_64 (custom kernel to fix Logitech dongle issue)
Version             : #1 SMP PREEMPT_DYNAMIC Thu Oct  5 19:57:47 UTC 2023
machine             : x86_64
processor           : 
byteorder           : little
LC_ALL              : None
LANG                : C.UTF-8
LOCALE              : en_US.UTF-8

pandas              : 2.1.1
numpy               : 1.26.0
pytz                : 2023.3.post1
dateutil            : 2.8.2
setuptools          : 65.5.1
pip                 : 22.3.1
Cython              : None
pytest              : 7.4.2
hypothesis          : None
sphinx              : None
blosc               : None
feather             : None
xlsxwriter          : None
lxml.etree          : None
html5lib            : 1.1
pymysql             : None
psycopg2            : None
jinja2              : None
IPython             : None
pandas_datareader   : None
bs4                 : None
bottleneck          : None
dataframe-api-compat: None
fastparquet         : None
fsspec              : None
gcsfs               : None
matplotlib          : None
numba               : None
numexpr             : None
odfpy               : None
openpyxl            : None
pandas_gbq          : None
pyarrow             : None
pyreadstat          : None
pyxlsb              : None
s3fs                : None
scipy               : None
sqlalchemy          : None
tables              : None
tabulate            : None
xarray              : None
xlrd                : None
zstandard           : None
tzdata              : 2023.3
qtpy                : None
pyqt5               : None

[davidboweninrupt]

Problem does not occur in pandas-2.2.0.dev0+329.g40f219ccf7 but of course 2.2 isn't available to people.

[paulreece]

It's not occurring on the main development branch as you stated.

I'm wondering if a fix needs to be back-ported for this.

[davidboweninrupt]

Just so you know, I've been able to pin to 2.1.0 and have that working with Python 3.12 by using a much larger Python image.

pandas 2.1.1 was a much smoother install on Python 3.12-slim due to no hiccups with gcc. That is what prompted me to move to 2.1.1.

As a result I don't need this back-ported, but it would be nice to have. I thought you might want to know about it too.

[paulreece]

As a result I don't need this back-ported, but it would be nice to have. I thought you might want to know about it too.

Sure but other folks might want to use this without Errors in Pandas 2.1.1. 😎

[jorisvandenbossche]

Under the hood, it is pd.cut that gets called, and we can also reproduce this directly by passing it an array instead of a series:

In [11]: ser = pd.Series([datetime.datetime.now(datetime.UTC)])

In [12]: pd.cut(ser, bins=2)
Out[12]: 
0    (2023-09-24 20:33:24.644781824, 2023-10-14 12:...
dtype: category
Categories (2, interval[datetime64[ns, UTC], right]): [ <
                                                       (2023-09-24 20:33:24.644781824, 2023-10-14 12:... <
                                                       (2023-10-14 12:01:29.534316032, 2023-11-03 03:...]

In [13]: pd.cut(ser.array, bins=2)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[13], line 1
----> 1 pd.cut(ser.array, bins=2)

File ~/miniconda3/envs/pandas21/lib/python3.11/site-packages/pandas/core/reshape/tile.py:247, in cut(x, bins, right, labels, retbins, precision, include_lowest, duplicates, ordered)
    245 original = x
    246 x = _preprocess_for_cut(x)
--> 247 x, dtype = _coerce_to_type(x)
    249 if not np.iterable(bins):
    250     if is_scalar(bins) and bins < 1:

File ~/miniconda3/envs/pandas21/lib/python3.11/site-packages/pandas/core/reshape/tile.py:505, in _coerce_to_type(x)
    501     x = x.to_numpy(dtype=np.float64, na_value=np.nan)
    503 if dtype is not None:
    504     # GH 19768: force NaT to NaN during integer conversion
--> 505     x = np.where(x.notna(), x.view(np.int64), np.nan)
    507 return x, dtype

AttributeError: 'DatetimeArray' object has no attribute 'notna'

So looking at the code there in cut, there was a recent change to ensure x was always converted to an Index (which would have this method), in #54919. So that might be the reason it is working now on the main branch.

And I suppose it is #54884 that actually caused the regression between 2.1.0 and 2.1.1

[jorisvandenbossche]

(btw @paulreece, if something is a regression that you could confirm, feel free to add the "next bug fix release" milestone to the issue)

[paulreece]

Will do that next time @jorisvandenbossche . Thanks for clarifying what was happening here!

[lithomas1]

Looks like this works on main for me now.

Hopefully this is fixed in 2.2 as well.
(I'll leave the issue open just in case it isn't)

[akivashor]

@phofl I would like to contribute since the pandas library was very useful for me in the last years. Can you assign this to me?

[sunwooma]

Hey akivashor, I am also interested in working on this issue. Do you want to help each other out?

[jasonmokk]

take