Raise error when datetime64 or timedelta64 values that are outside the valid range for ns precision are converted to ns precision

[andrewpauling]

Use _possibly_convert_obhects to raise an error when converting datetime64 or timedelta64 objects that are in some units other than ns to ns as per pandas requirements, resulting in dates outside the valid range for ns precision being silently changed to incorrect values.

• Closes assign_coords with datetime64[us] changes dtype to datetime64[ns] #4427
• Passes isort . && black . && mypy . && flake8
• User visible changes (including notable bug fixes) are documented in whats-new.rst

[max-sixty]

Thanks @andrewpauling , this looks good.

Could you add a test?

[andrewpauling]

Hi @max-sixty, I have just added what I hope is a good test, I haven't written one before, but tried to base it on others in the test_variable file

[max-sixty]

It looks great! I made one suggestion.

Any thoughts from others?

[spencerkclark]

Nice! Could you also add a similar test for timedelta64 data?

[andrewpauling]

Nice! Could you also add a similar test for timedelta64 data?

Hi @spencerkclark, I have been trying to work on a test for timedelta64 data but I don't think I understand timedelta data very well. I can't seem to come up with an example that raises an OutOfBoundsTimedelta error. From what I found online (https://pandas.pydata.org/pandas-docs/stable/user_guide/timedeltas.html), the valid range for nanosecond precision should be about 100000 days. When I try to input an np.timedelta64[D] object of more days than that, it just overflows and gives nonsense without throwing an error. Do you have any suggestions?

[spencerkclark]

You're absolutely right @andrewpauling -- pandas currently doesn't seem to have overflow protection in the case of casting timedelta64 types. I wrongly assumed that it would! I went ahead and opened an issue there: pandas-dev/pandas#36615. We'll see what they say.

For now maybe you can add the test and attach an expected failure decorator to it?

In researching this, I came across numpy/numpy#16352; if that eventually gets addressed this might all be moot.

[andrewpauling]

Thanks @spencerkclark, I have added a test with the decorator you mentioned and edited the formatting in the whats-new file as above. Hopefully it should be good now.

[max-sixty]

Excellent, thanks @andrewpauling ! LGTM

[spencerkclark]

Thanks @andrewpauling! Sorry for taking a bit to re-review; it looks like there's now a minor merge conflict in the what's new page. Once that's sorted out, I think this should be good to go.

[max-sixty]

Great, let's merge on green!

[andrewpauling]

Cool! Thank you all for your help.

[spencerkclark]

Thanks again @andrewpauling!

[max-sixty]

Thanks a lot @andrewpauling !