ENH: overflow-safe astype for datetime64/timedelta64 unit conversion

[jbrockmendel]

import numpy as np
from pandas._libs.tslibs.conversion import ensure_datetime64ns
arr = np.array(["2367-12-31 12:00:00"], dtype="datetime64[h]")

>>> arr.astype("datetime64[ns]")
array(['1783-06-11T12:25:26.290448384'], dtype='datetime64[ns]')

>>> ensure_datetime64ns(arr)
[...]
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 2367-12-31 12:00:00

Is there a compelling reason numpy doesnt do this?

[seberg]

No, we should raise it. It is a bit clumsy to raise errors from within the casting machinery currently, I think that is pretty much the only reason...

EDIT: There could be a point of just returning NaT, although not really better IMO.

[jbrockmendel]

It is a bit clumsy to raise errors from within the casting machinery currently

No idea how to be helpful on that front, but here are links to the relevant pandas code that may be upstream-able:

https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/conversion.pyx#L155
https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/np_datetime.pyx#L95
https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/src/datetime/np_datetime.c#L250

[jbrockmendel]

In addition to astype, it'd be nice if addition/subtraction were overflow-safe

np.seterr(over="raise")

arr = np.arange(10)*10**18
arr2 = arr.view("datetime64[ns]")
arr3 = arr.view("timedelta64[ns]")

>>> arr2 + arr3
array(['1970-01-01T00:00:00.000000000', '2033-05-18T03:33:20.000000000',
       '2096-10-02T07:06:40.000000000', '2160-02-18T10:40:00.000000000',
       '2223-07-06T14:13:20.000000000', '1702-05-02T18:12:06.290448384',
       '1765-09-16T21:45:26.290448384', '1829-02-02T01:18:46.290448384',
       '1892-06-19T04:52:06.290448384', '1955-11-05T08:25:26.290448384'],
      dtype='datetime64[ns]')

[jbrockmendel]

Just implemented a version of this in pandas-dev/pandas#46478 that I'm hoping we can upstream to numpy somehow.