CF: also decode time bounds when available #2571

fmaussion · 2018-11-24T16:50:13Z

Closes CF conventions: time_bnds and time units #2565
Tests added
Fully documented, including whats-new.rst for all changes and api.rst for new API

Not sure if this is the best way to handle it, but it seems to work

pep8speaks · 2018-11-24T16:50:15Z

Hello @fmaussion! Thanks for submitting the PR.

There are no PEP8 issues in the file xarray/conventions.py !
There are no PEP8 issues in the file xarray/tests/test_coding_times.py !

spencerkclark

Thanks for putting this together @fmaussion! This seems like the right place to handle this.

One minor design question here is: how strictly do we want to adhere to roundtripping variable attributes in Datasets? With this implementation, if one decodes the times, and then saves things back out to a file, a "time bounds" variable will contain units and calendar attributes even though it might not have them initially. I'm not sure if this is a big deal (in this case it is simply adding redundant metadata), but I just wanted to bring it up in case it was a concern.

spencerkclark · 2018-11-24T17:46:34Z

xarray/tests/test_coding_times.py

@@ -624,6 +624,62 @@ def test_decode_cf(calendar):
            assert ds.test.dtype == np.dtype('M8[ns]')


+def test_decode_cf_time_bounds():
+
+    da = DataArray(np.arange(6).reshape((3, 2)), coords={'time': [1, 2, 3]},


Might it make sense to make this DataArray a fixture and split these cases into separate tests?

spencerkclark · 2018-11-24T19:03:06Z

xarray/conventions.py

+                    new_attrs['calendar'] = attrs['calendar']
+                to_update[attrs['bounds']] = new_attrs
+        # For all bound variables, update their time attribute if not present
+        for k, new_attrs in to_update.items():


I think we could update the attributes of the potential bounds variables immediately within the for v in variables.values() loop rather than use another loop here (this would remove the need for the to_update dictionary).

spencerkclark · 2018-11-24T19:09:37Z

xarray/conventions.py

@@ -350,6 +350,25 @@ def stackable(dim):
        drop_variables = []
    drop_variables = set(drop_variables)

+    # Time bounds coordinates might miss the decoding attributes
+    # https://github.com/pydata/xarray/issues/2565


Would it also make sense to add a comment with the URL for the "Cell Boundaries" section of the CF conventions here?

spencerkclark · 2018-11-24T19:13:32Z

xarray/conventions.py

+        to_update = dict()
+        for v in variables.values():
+            attrs = v.attrs
+            if ('units' in attrs and 'since' in attrs['units'] and


It might be cleaner to write something like:

has_date_units = 'units' in attrs and 'since' in attrs['units'] if has_date_units and 'bounds' in attrs:

shoyer · 2018-11-25T03:34:46Z

Could we move this logic into a separate helper function? That would make things a little better organized and easier to test.

fmaussion · 2018-11-29T23:00:10Z

I addressed all comments except the fixtures, which seemed a bit overkill (but I simplified the tests). Thanks!

spencerkclark

Thanks @fmaussion, just a minor optional naming suggestion, otherwise this looks good to me!

spencerkclark · 2018-11-30T20:34:59Z

xarray/conventions.py

+        has_date_units = 'units' in attrs and 'since' in attrs['units']
+        if has_date_units and 'bounds' in attrs:
+            if attrs['bounds'] in variables:
+                to_update = variables[attrs['bounds']].attrs


Perhaps a clearer name would be bounds_attrs?

Suggested change

to_update = variables[attrs['bounds']].attrs

bounds_attrs = variables[attrs['bounds']].attrs

fmaussion · 2018-12-01T11:51:50Z

Thanks! Made the change as requested. Regarding the general design:

With this implementation, if one decodes the times, and then saves things back out to a file, a "time bounds" variable will contain units and calendar attributes even though it might not have them initially. I'm not sure if this is a big deal (in this case it is simply adding redundant metadata), but I just wanted to bring it up in case it was a concern.

I agree - I don't think it is a big deal either. It must also be noted that this might break some code in downstream libraries (including mine) which has built workarounds for this and will now error because the bounds are already decoded. Here also, I'm quite confident that this is an edge case but its worth mentioning. Should I put my what's new in "Breaking changes" instead?

spencerkclark · 2018-12-02T02:10:22Z

It must also be noted that this might break some code in downstream libraries (including mine) which has built workarounds for this and will now error because the bounds are already decoded. Here also, I'm quite confident that this is an edge case but its worth mentioning. Should I put my what's new in "Breaking changes" instead?

I for sure see this perspective. I also think a plausible case could be made that this change is like a "bug fix," that is something that people may have needed to work around before in various ways, but ultimately should not have needed to. So I think it's up to you what you think is best.

If you do decide to shift it to the breaking changes section, I would suggest being a little more specific about under what circumstances the behavior is changing (i.e. this only impacts the behavior for time bounds variables defined via CF conventions that do not already have "units" and "calendar" attributes).

shoyer · 2018-12-12T16:58:11Z

+1 for putting this in "Breaking changes" (we try to be pretty conservative with the definition of a strict "Enhancement"), but otherwise this looks good to me.

fmaussion · 2018-12-16T23:40:32Z

+1 for putting this in "Breaking changes"

Sorry for being so slow. I just done it, will merge tomorrow once the tests pass.

* master: DEP: drop python 2 support and associated ci mods (pydata#2637) TST: silence warnings from bottleneck (pydata#2638) revert to dev version DOC: fix docstrings and doc build for 0.11.1 Source encoding always set when opening datasets (pydata#2626) Add flake check to travis (pydata#2632) Fix dayofweek and dayofyear attributes from dates generated by cftime_range (pydata#2633) silence import warning (pydata#2635) fill_value in shift (pydata#2470) Flake fixed (pydata#2629) Allow passing of positional arguments in `apply` for Groupby objects (pydata#2413) Fix failure in time encoding for pandas < 0.21.1 (pydata#2630) Fix multiindex selection (pydata#2621) Close files when CachingFileManager is garbage collected (pydata#2595) added some logic to deal with rasterio objects in addition to filepaths (pydata#2589) Get 0d slices of ndarrays directly from indexing (pydata#2625) FIX Don't raise a deprecation warning for xarray.ufuncs.{angle,iscomplex} (pydata#2615) CF: also decode time bounds when available (pydata#2571)

CF: also decode time bounds when available

b0c3dcf

Fix failing test when cftime not present and what's new

de5865d

fmaussion requested a review from spencerkclark November 24, 2018 17:39

Fix windows

e40861a

spencerkclark reviewed Nov 24, 2018

View reviewed changes

fmaussion added 2 commits November 29, 2018 22:25

Reviews

8240be0

Merge branch 'master' into fix-docs

6d529c1

fmaussion mentioned this pull request Nov 30, 2018

Doc: example run_with_spinup OGGM/oggm#610

Closed

spencerkclark approved these changes Dec 1, 2018

View reviewed changes

Reviews 2

326f7ae

Merge branch 'master' into fix-docs

0998d2b

dcherian merged commit 57348ab into pydata:master Dec 19, 2018

spencerkclark mentioned this pull request Mar 20, 2019

time decoding error with "days since" #521

Closed

dcherian mentioned this pull request May 7, 2019

to_netcdf with decoded time can create file with inconsistent time:units and time_bounds:units #2921

Closed

rabernat mentioned this pull request Sep 9, 2020

netCDF time encoding challenges mom-ocean/MOM6#1203

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CF: also decode time bounds when available #2571

CF: also decode time bounds when available #2571

fmaussion commented Nov 24, 2018 •

edited

pep8speaks commented Nov 24, 2018

spencerkclark left a comment •

edited

spencerkclark Nov 24, 2018

spencerkclark Nov 24, 2018

spencerkclark Nov 24, 2018

spencerkclark Nov 24, 2018

shoyer commented Nov 25, 2018

fmaussion commented Nov 29, 2018 •

edited

spencerkclark left a comment

spencerkclark Nov 30, 2018

fmaussion commented Dec 1, 2018

spencerkclark commented Dec 2, 2018

shoyer commented Dec 12, 2018

fmaussion commented Dec 16, 2018

	to_update = variables[attrs['bounds']].attrs
	bounds_attrs = variables[attrs['bounds']].attrs

CF: also decode time bounds when available #2571

CF: also decode time bounds when available #2571

Conversation

fmaussion commented Nov 24, 2018 • edited

pep8speaks commented Nov 24, 2018

spencerkclark left a comment • edited

Choose a reason for hiding this comment

spencerkclark Nov 24, 2018

Choose a reason for hiding this comment

spencerkclark Nov 24, 2018

Choose a reason for hiding this comment

spencerkclark Nov 24, 2018

Choose a reason for hiding this comment

spencerkclark Nov 24, 2018

Choose a reason for hiding this comment

shoyer commented Nov 25, 2018

fmaussion commented Nov 29, 2018 • edited

spencerkclark left a comment

Choose a reason for hiding this comment

spencerkclark Nov 30, 2018

Choose a reason for hiding this comment

fmaussion commented Dec 1, 2018

spencerkclark commented Dec 2, 2018

shoyer commented Dec 12, 2018

fmaussion commented Dec 16, 2018

fmaussion commented Nov 24, 2018 •

edited

spencerkclark left a comment •

edited

fmaussion commented Nov 29, 2018 •

edited