BUG: Future time decoding

[znicholls]

• Tests added
• Passes pre-commit run --all-files
• User visible changes (including notable bug fixes) are documented in whats-new.rst

This identifies (and hopefully soon fixes) an issue with decoding times that are way into the future. Without re-introducing a change made in #4684, the test fails (see https://github.com/pydata/xarray/pull/5050/checks?check_run_id=2136824754). With the change put back in, the test passes (see https://github.com/pydata/xarray/pull/5050/checks?check_run_id=2136892412).

@spencerkclark can I ask for your help on this one please? It seems that the removal in #4684 caused this case (which wasn't tested) to break in v0.17 and it's unclear to me where the fix should be (I'm guessing pandas from the discussions in #4684...).

[spencerkclark]

@znicholls oof this was an oversight on my part. I naively took the tests passing and the fact that the upstream issue in pandas had been fixed to mean that we did not need this anymore. As you noted it turns out we still do.

Basically the issue stems from this optimization:

xarray/xarray/coding/times.py

Lines 190 to 194 in a6f51c6

	# Cast input ordinals to integers of nanoseconds because pd.to_timedelta
	# works much faster when dealing with integers (GH 1399).
	flat_num_dates_ns_int = (flat_num_dates * _NS_PER_TIME_DELTA[delta]).astype(
	np.int64
	)

If the encoded values are too large to be represented with a timedelta64[ns] type, this line does not catch it; instead it silently overflows:

In [1]: import xarray as xr; import pandas as pd; import numpy as np

In [2]: num_dates = np.array([164375])

In [3]: units = "days since 1850-01-01 00:00:00"

In [4]: calendar = "proleptic_gregorian"

In [5]: (num_dates * xr.coding.times._NS_PER_TIME_DELTA["D"]).astype(np.int64)
Out[5]: array([-4244744073709551616])

So this is indeed the correct place to fix this (since it has to do with our bespoke decoding logic and not to do with pandas). If you can just add a what's new entry, I think this is ready to go!

[znicholls]

If you can just add a what's new entry, I think this is ready to go!

Done, thanks!

[spencerkclark]

In the meantime it occurs to me that if you (or anyone else experiencing this issue) need a workaround, you should be able to set use_cftime=True in calls to open_dataset or decode_cf, and I think things should work (in that case it won't even try decoding the times with pandas).

[spencerkclark]

Thanks @znicholls!