BUG: IndexError with pandas.DataFrame.cumsum where dtype=timedelta64[ns]

[omor1]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

# Example DataFrame from pandas.DataFrame.cumsum documentation
df = pd.DataFrame([[2.0, 1.0],
                   [3.0, np.nan],
                   [1.0, 0.0]],
                  columns=list('AB'))

# IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed
df.astype('timedelta64[ns]').cumsum()

# These are fine
df.cumsum().astype('timedelta64[ns]')
df.apply(lambda s: s.astype('timedelta64[ns]').cumsum())
df.astype('timedelta64[ns]').apply(lambda s: s.cumsum())

Issue Description

A cumulative sum along the columns of a DataFrame with timedelta64[ns] types results in an IndexError. Cumulative min and max work correctly, and product correctly results in a TypeError, since it's not supported for timedelta.

Cumulative sums work correctly for Series with timedelta64[ns] type, so this seems to be a bug in DataFrame.

This is perhaps related to #41720?

Expected Behavior

pandas.DataFrame.cumsum on a DataFrame with timedelta64[ns] columns should not erroneously raise IndexError and instead compute the cumulative sum, with the same effect as df.apply(lambda s: s.cumsum()).

Installed Versions

Installed Versions

INSTALLED VERSIONS
------------------
commit                : bdc79c146c2e32f2cab629be240f01658cfb6cc2
python                : 3.12.2.final.0
python-bits           : 64
OS                    : Darwin
OS-release            : 23.4.0
Version               : Darwin Kernel Version 23.4.0: Wed Feb 21 21:45:48 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T8122
machine               : arm64
processor             : arm
byteorder             : little
LC_ALL                : None
LANG                  : en_US.UTF-8
LOCALE                : en_US.UTF-8

pandas                : 2.2.1
numpy                 : 1.26.4
pytz                  : 2024.1
dateutil              : 2.9.0.post0
setuptools            : None
pip                   : 24.0
Cython                : None
pytest                : None
hypothesis            : None
sphinx                : None
blosc                 : None
feather               : None
xlsxwriter            : None
lxml.etree            : None
html5lib              : None
pymysql               : None
psycopg2              : None
jinja2                : None
IPython               : 8.22.2
pandas_datareader     : None
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : None
bottleneck            : None
dataframe-api-compat  : None
fastparquet           : None
fsspec                : None
gcsfs                 : None
matplotlib            : 3.8.3
numba                 : None
numexpr               : None
odfpy                 : None
openpyxl              : None
pandas_gbq            : None
pyarrow               : None
pyreadstat            : None
python-calamine       : None
pyxlsb                : None
s3fs                  : None
scipy                 : 1.12.0
sqlalchemy            : None
tables                : None
tabulate              : None
xarray                : None
xlrd                  : None
zstandard             : None
tzdata                : 2024.1
qtpy                  : None
pyqt5                 : None

[pmhatre1]

The code seems to be taking an single list instead of numpy array below

[pmhatre1]

original_list = func(y)
result = [[x] for x in original_list]
result = np.array(result)

This might help?

[Aloqeely]

The problem exists in core\array_algos\datetimelike_accumulations.py, _cum_func:

result = func(y)
result[mask] = iNaT

Essentially, what this is doing is performing func (np.cumsum in this case) on your whole data array, and by default np.cumsum computes the sum over a flattened array and returns it, but then when masking the NA values, it is expecting that the data array was in it's original shape and not flattened.

To fix this, you have to pass axis as 0 to func (again, np.cumsum in this case), so that it performs the cumsum along the given axis and not over the whole flattened array (it's the only func with axis=None by default, unlike cummin and cummax which have axis=0 by default and work properly)

I'd like to try fixing this issue, but it would be helpful to understand why the developers chose not to pass axis to func and instead handle its behavior internally.

And to explain why your other examples worked, in the first one, it did not use _cum_func function, as the values were converted to a timedelta64[ns] after computing the cumsum, but in the second and third examples, it was handled well and only 1 column at a time was passed to _cum_func (I couldn't trace back to how that was happening)

[pmhatre1]

@phofl @mroeschke any comment on this based on #41720. I found result is 1D while mask is an array. Thus I tried

flattened_mask = mask.flatten()
result[flattened_mask] = iNaT

but ended up in error at

block = self.blocks[self.blknos[i]]
values = block.iget(self.blklocs[i])
bp = BlockPlacement(slice(0, len(values)))

    in managers.py

[Aloqeely]

take

[pmhatre1]

Thanks @Aloqeely for helping me understand this.