OverflowError time values outside range of 64 bit signed integers

[durack1]

I am a little confused by this error, I am opening a CMIP3 netcdf file with what appears to be valid data, however, cftime is throwing errors with what appear to be numbers that are within the 64 bit signed integer range.

The file is a 1.1GiB file downloadable from here

Below is the example reproducing the error:

Python 3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 07:04:59) [GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xarray as xr
>>> xr.__version__
'2022.6.0'
>>> import cftime as cft
>>> cft.__version__
'1.6.1'
>>> ds = xr.open_dataset("rhopoto_O1.PIcntrl_2.CCSM.ocnm.0585-01_cat_0589-12.nc", use_cftime=True)
Traceback (most recent call last):
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/coding/times.py", line 180, in _decode_cf_datetime_dtype
    result = decode_cf_datetime(example_value, units, calendar, use_cftime)
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/coding/times.py", line 293, in decode_cf_datetime
    dates = _decode_datetime_with_cftime(flat_num_dates, units, calendar)
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/coding/times.py", line 201, in _decode_datetime_with_cftime
    cftime.num2date(num_dates, units, calendar, only_use_cftime_datetimes=True)
  File "src/cftime/_cftime.pyx", line 586, in cftime._cftime.num2date
  File "src/cftime/_cftime.pyx", line 385, in cftime._cftime.cast_to_int
OverflowError: time values outside range of 64 bit signed integers

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/backends/api.py", line 531, in open_dataset
    backend_ds = backend.open_dataset(
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/backends/netCDF4_.py", line 569, in open_dataset
    ds = store_entrypoint.open_dataset(
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/backends/store.py", line 29, in open_dataset
    vars, attrs, coord_names = conventions.decode_cf_variables(
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/conventions.py", line 521, in decode_cf_variables
    new_vars[k] = decode_cf_variable(
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/conventions.py", line 369, in decode_cf_variable
    var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/coding/times.py", line 682, in decode
    dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/coding/times.py", line 190, in _decode_cf_datetime_dtype
    raise ValueError(msg)
ValueError: unable to decode time units 'days since 0000-1-1' with "calendar 'noleap'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.
>>> ds = xr.open_dataset("rhopoto_O1.PIcntrl_2.CCSM.ocnm.0585-01_cat_0589-12.nc", decode_times=False)
>>> ds.time
<xarray.DataArray 'time' (time: 60)>
array([213540.5, 213570. , 213599.5, 213630. , 213660.5, 213691. , 213721.5,
       213752.5, 213783. , 213813.5, 213844. , 213874.5, 213905.5, 213935. ,
       213964.5, 213995. , 214025.5, 214056. , 214086.5, 214117.5, 214148. ,
       214178.5, 214209. , 214239.5, 214270.5, 214300. , 214329.5, 214360. ,
       214390.5, 214421. , 214451.5, 214482.5, 214513. , 214543.5, 214574. ,
       214604.5, 214635.5, 214665. , 214694.5, 214725. , 214755.5, 214786. ,
       214816.5, 214847.5, 214878. , 214908.5, 214939. , 214969.5, 215000.5,
       215030. , 215059.5, 215090. , 215120.5, 215151. , 215181.5, 215212.5,
       215243. , 215273.5, 215304. , 215334.5])
Coordinates:
  * time     (time) float64 2.135e+05 2.136e+05 ... 2.153e+05 2.153e+05
Attributes:
    calendar:       noleap
    standard_name:  time
    axis:           T
    units:          days since 0000-1-1
    bounds:         time_bnds
>>>

I have also tried to fix the quirky calendar "days since 0000-1-1" -> "days since 0001-01-01" but this doesn't seem to get me any further:

>>> ds.time.attrs["units"] = "days since 0001-01-01"
>>> ds.time
<xarray.DataArray 'time' (time: 60)>
array([213540.5, 213570. , 213599.5, 213630. , 213660.5, 213691. , 213721.5,
       213752.5, 213783. , 213813.5, 213844. , 213874.5, 213905.5, 213935. ,
       213964.5, 213995. , 214025.5, 214056. , 214086.5, 214117.5, 214148. ,
       214178.5, 214209. , 214239.5, 214270.5, 214300. , 214329.5, 214360. ,
       214390.5, 214421. , 214451.5, 214482.5, 214513. , 214543.5, 214574. ,
       214604.5, 214635.5, 214665. , 214694.5, 214725. , 214755.5, 214786. ,
       214816.5, 214847.5, 214878. , 214908.5, 214939. , 214969.5, 215000.5,
       215030. , 215059.5, 215090. , 215120.5, 215151. , 215181.5, 215212.5,
       215243. , 215273.5, 215304. , 215334.5])
Coordinates:
  * time     (time) float64 2.135e+05 2.136e+05 ... 2.153e+05 2.153e+05
Attributes:
    calendar:       noleap
    standard_name:  time
    axis:           T
    units:          days since 0001-01-01
    bounds:         time_bnds
>>> ds = xr.decode_cf(ds)
Traceback (most recent call last):
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/coding/times.py", line 270, in decode_cf_datetime
    dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar)
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/coding/times.py", line 207, in _decode_datetime_with_pandas
    raise OutOfBoundsDatetime(
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Cannot decode times from a non-standard calendar, 'noleap', using pandas.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/coding/times.py", line 180, in _decode_cf_datetime_dtype
    result = decode_cf_datetime(example_value, units, calendar, use_cftime)
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/coding/times.py", line 272, in decode_cf_datetime
    dates = _decode_datetime_with_cftime(
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/coding/times.py", line 201, in _decode_datetime_with_cftime
    cftime.num2date(num_dates, units, calendar, only_use_cftime_datetimes=True)
  File "src/cftime/_cftime.pyx", line 586, in cftime._cftime.num2date
  File "src/cftime/_cftime.pyx", line 385, in cftime._cftime.cast_to_int
OverflowError: time values outside range of 64 bit signed integers

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/conventions.py", line 655, in decode_cf
    vars, attrs, coord_names = decode_cf_variables(
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/conventions.py", line 521, in decode_cf_variables
    new_vars[k] = decode_cf_variable(
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/conventions.py", line 369, in decode_cf_variable
    var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/coding/times.py", line 682, in decode
    dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
  File "~/mambaforge/envs/xcd031spy532mat353/lib/python3.10/site-packages/xarray/coding/times.py", line 190, in _decode_cf_datetime_dtype
    raise ValueError(msg)
ValueError: unable to decode time units 'days since 0001-01-01' with "calendar 'noleap'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.
>>>

I'd appreciate a heads up about how to solve this issue, it seems like it should work - does for adjacent (earlier and later timesteps) in the same dataset.

[spencerkclark]

Interesting. I can decode those times in my environment. It looks like maybe there is an associated time_bnds variable. Could you show what the values of that DataArray look like (if it indeed exists)? It's possible values there could be tripping up the decoding.

[durack1]

@spencerkclark looks like you've nailed it:

>>> ds.time_bnds
<xarray.DataArray 'time_bnds' (time: 60, bnds: 2)>
array([[2.13525e+05, 2.13556e+05],
       [2.13556e+05, 2.13584e+05],
       [2.13584e+05, 2.13615e+05],
       [2.13615e+05, 2.13645e+05],
       [2.13645e+05, 2.13676e+05],
       [2.13676e+05, 2.13706e+05],
       [2.13706e+05, 2.13737e+05],
       [2.13737e+05, 2.13768e+05],
       [2.13768e+05, 2.13798e+05],
       [2.13798e+05, 2.13829e+05],
       [2.13829e+05, 2.13859e+05],
       [2.13859e+05, 2.13890e+05],
       [2.13890e+05, 2.13921e+05],
       [2.13921e+05, 2.13949e+05],
       [2.13949e+05, 2.13980e+05],
       [2.13980e+05, 2.14010e+05],
       [2.14010e+05, 2.14041e+05],
       [2.14041e+05, 2.14071e+05],
       [2.14071e+05, 2.14102e+05],
       [2.14102e+05, 2.14133e+05],
       [2.14133e+05, 2.14163e+05],
       [2.14163e+05, 2.14194e+05],
       [2.14194e+05, 2.14224e+05],
       [2.14224e+05, 2.14255e+05],
       [2.14255e+05, 2.14286e+05],
       [2.14286e+05, 2.14314e+05],
       [2.14314e+05, 2.14345e+05],
       [2.14345e+05, 2.14375e+05],
       [2.14375e+05, 2.14406e+05],
       [2.14406e+05, 2.14436e+05],
       [2.14436e+05, 2.14467e+05],
       [2.14467e+05, 2.14498e+05],
       [2.14498e+05, 2.14528e+05],
       [2.14528e+05, 2.14559e+05],
       [2.14559e+05, 2.14589e+05],
       [2.14589e+05, 2.14620e+05],
       [2.14620e+05, 2.14651e+05],
       [2.14651e+05, 2.14679e+05],
       [2.14679e+05, 2.14710e+05],
       [2.14710e+05, 2.14740e+05],
       [2.14740e+05, 2.14771e+05],
       [2.14771e+05, 2.14801e+05],
       [2.14801e+05, 2.14832e+05],
       [2.14832e+05, 2.14863e+05],
       [2.14863e+05, 2.14893e+05],
       [2.14893e+05, 2.14924e+05],
       [2.14924e+05, 2.14954e+05],
       [2.14954e+05, 2.14985e+05],
       [2.14985e+05, 2.15016e+05],
       [2.15016e+05, 2.15044e+05],
       [2.15044e+05, 2.15075e+05],
       [2.15075e+05, 2.15105e+05],
       [2.15105e+05, 2.15136e+05],
       [2.15136e+05, 2.15166e+05],
       [2.15166e+05, 2.15197e+05],
       [2.15197e+05, 2.15228e+05],
       [2.15228e+05, 2.15258e+05],
       [2.15258e+05, 2.15289e+05],
       [2.15289e+05, 2.15319e+05],
       [9.96921e+36, 9.96921e+36]])
Coordinates:
  * time     (time) float64 2.135e+05 2.136e+05 ... 2.153e+05 2.153e+05
Dimensions without coordinates: bnds
>>>

[durack1]

It'd be great, if the error message could be improved to note the issue is with time_bnds.

I have another edge case that tripped over cftime, will open another issue - it's a very very edge case

[spencerkclark]

It'd be great, if the error message could be improved to note the issue is with time_bnds.

I totally agree -- I opened an issue in xarray to track this (pydata/xarray#7145). It will need to be addressed there.

[durack1]

@spencerkclark perfect, thanks.

How would you recommend this issue is worked around, I have played with ds.drop(["time_bnds") but it seems that the same issue occurs when calling xr.decode_cf(ds) immediately after

[spencerkclark]

Are there other time-like variables in the Dataset that have problematic values? That would be my guess if dropping "time_bounds" alone doesn't work.

[spencerkclark]

Just to make sure, are you calling drop like this (i.e. re-assigning what it returns to ds)?

ds = ds.drop(["time_bounds"])

Note that drop does not operate in-place, so without re-assignment, ds will still contain the "time_bounds" variable.

[durack1]

@spencerkclark good catch, I have corrected my code to now be ds = ds.drop_vars(["time_bnds"]), but I am still hitting issues, not sure why. An ncdump -h output is below:

$ ncdump -h rhopoto_O1.PIcntrl_2.CCSM.ocnm.0585-01_cat_0589-12.nc
netcdf rhopoto_O1.PIcntrl_2.CCSM.ocnm.0585-01_cat_0589-12 {
dimensions:
        lon = 320 ;
        lat = 395 ;
        depth = 40 ;
        bnds = 2 ;
        time = UNLIMITED ; // (60 currently)
variables:
        double lon_bnds(lon, bnds) ;
        double lat_bnds(lat, bnds) ;
        double depth_bnds(depth, bnds) ;
        double time_bnds(time, bnds) ;
        double lon(lon) ;
                lon:axis = "X" ;
                lon:standard_name = "longitude" ;
                lon:bounds = "lon_bnds" ;
                lon:long_name = "Longitude" ;
                lon:units = "degrees_east" ;
        double lat(lat) ;
                lat:axis = "Y" ;
                lat:standard_name = "latitude" ;
                lat:bounds = "lat_bnds" ;
                lat:long_name = "Latitude" ;
                lat:units = "degrees_north" ;
        double depth(depth) ;
                depth:axis = "Z" ;
                depth:standard_name = "depth" ;
                depth:positive = "down" ;
                depth:units = "m" ;
                depth:bounds = "depth_bnds" ;
        double time(time) ;
                time:calendar = "noleap" ;
                time:standard_name = "time" ;
                time:axis = "T" ;
                time:units = "days since 0000-1-1" ;
                time:bounds = "time_bnds" ;
        float rhopoto(time, depth, lat, lon) ;
                rhopoto:comment = "Created using NCL code CCSM_ocnm_2cf.ncl on\n",
                        " machine tempest" ;
                rhopoto:missing_value = 1.e+20f ;
                rhopoto:long_name = "sea_water_potential_density" ;
                rhopoto:cell_methods = "time: mean (interval: 1 month)" ;
                rhopoto:history = "Interpolated to regular grid from dipole grid,\n",
                        "PD*1.e3" ;
                rhopoto:units = "kg m-3" ;
                rhopoto:original_units = "g cm-3" ;
                rhopoto:original_name = "PD" ;
                rhopoto:standard_name = "sea_water_potential_density" ;
                rhopoto:_FillValue = 1.e+20f ;

// global attributes:
                :table_id = "Table O1" ;
...

[spencerkclark]

Could you show the full error message? I downloaded the file, and successfully ran the following code:

>>> import xarray as xr
>>> ds = xr.open_dataset("~/Downloads/rhopoto_O1.PIcntrl_2.CCSM.ocnm.0585-01_cat_0589-12.nc", decode_times=False)
>>> ds = ds.drop_vars(["time_bnds"])
>>> xr.decode_cf(ds)
<xarray.Dataset>
Dimensions:     (lon: 320, bnds: 2, lat: 395, depth: 40, time: 60)
Coordinates:
  * lon         (lon) float64 1.063 2.188 3.313 4.438 ... 357.7 358.8 359.9
  * lat         (lat) float64 -79.22 -78.69 -78.15 -77.62 ... 89.37 89.9 90.0
  * depth       (depth) float64 5.006 15.07 25.28 ... 5.125e+03 5.375e+03
  * time        (time) object 0585-01-16 12:00:00 ... 0589-12-16 12:00:00
Dimensions without coordinates: bnds
Data variables:
    lon_bnds    (lon, bnds) float64 ...
    lat_bnds    (lat, bnds) float64 ...
    depth_bnds  (depth, bnds) float64 ...
    rhopoto     (time, depth, lat, lon) float32 ...
Attributes: (12/13)
    table_id:        Table O1
    title:           model output prepared for IPCC AR4
    institution:     NCAR (National Center for Atmospheric \nResearch, Boulde...
    source:          CCSM3.0, version vector05 (2004): \natmosphere: CAM3.0, ...
    contact:         ccsm@ucar.edu
    project_id:      IPCC Fourth Assessment
    ...              ...
    references:      Collins, W.D., et al., 2005:\n The Community Climate Sys...
    acknowledgment:   Any use of CCSM data should acknowledge the contributio...
    realization:     2
    experiment_id:   pre-industrial control experiment
    history:         Created from CCSM3 case b30.020.ES02\n by ilana@ucar.edu...
    comment:         This simulation was initiated from year 430 of \n CCSM3 ...

An alternative if you'd like a one-liner for opening the Dataset, would be to use the drop_variables argument of open_dataset:

>>> ds = xr.open_dataset("~/Downloads/rhopoto_O1.PIcntrl_2.CCSM.ocnm.0585-01_cat_0589-12.nc", drop_variables=["time_bnds"])

[durack1]

I was trying to combine multiple steps in one operation (as I am looping over a lot of files), so ended up with:

filePath = "rhopoto_O1.PIcntrl_2.CCSM.ocnm.0585-01_cat_0589-12.nc",
...
ds = (
xr.open_dataset(filePath, decode_times=False)
.pipe(fixFunc(fixStr, fixStrInfo))
.pipe(xr.decode_cf)
)

fix is a function that deals with all the issues I encountered, specific for each dataset and for this file, it looks like:

fixStr = "ds.time.attrs['units'] = 'days since 0001-01-01'; ds = ds.drop(['time_bnds'])"
fixStrInfo = "fix bad time:units 0000-1-1"
...
def fixFunc(fixStr, fixStrInfo):
    def fix(ds):
        print(fixStrInfo)
        exec(fixStr)
        return ds
    return fix

And that's where I believe I have fallen over, the pipe and assigning the result to ds isn't going to work.

Your suggestion of appending , drop_variables=["time_bnds"]) gets me around the problem, thanks for helping this noob!

[jswhit]

Am I correct in assuming that no action is required (or desired) on the cftime side then?

[spencerkclark]

Yes, that's correct @jswhit. I have a PR open in xarray to address this: pydata/xarray#7147.

[jswhit]

OK, I'll close this one then