Added support for cftime types #2728

philippjfr · 2018-05-25T02:01:57Z

This PR adds support for cftime types which are a custom time type used by xarray for dates that cannot be stored in a datetime64 type. A small example:

ref_date = '0181-01-01'
ds = xr.DataArray([1, 2, 3], dims=['time'],
                  coords={'time': ('time', [1, 2, 3],
                                   {'units': 'days since %s' % ref_date})}
                  ).to_dataset(name='foo')
with xr.set_options(enable_cftimeindex=True):
    ds = xr.decode_cf(ds)
print(ds)
hv_ds = hv.Dataset(ds)
hv_ds.to(hv.Curve)

<xarray.Dataset>
Dimensions:  (time: 3)
Coordinates:
  * time     (time) object 0181-01-02 00:00:00 0181-01-03 00:00:00 ...
Data variables:
    foo      (time) int64 ...

This approach seems to work well but there are a questions I have, which @rabernat and @spencerclark hopefully can answer:

Which types does it make sense to support, and do they all support conversion to a timetuple? As far as I could tell this is the list of types cftime provides:

cftime._cftime.Datetime360Day
cftime._cftime.DatetimeGregorian
cftime._cftime.DatetimeJulian
cftime._cftime.DatetimeNoLeap
cftime._cftime.DatetimeProlepticGregorian

Secondly in what context can I expect these types to be used? Will they mostly be used by xarray or should I also handle them elsewhere?

Addresses Add support for cftime types #2709 and holoviews / bokeh doesn't like cftime coords pydata/xarray#2164
Add unit tests
Handle matplotlib
Handle datashader (if possible)

rabernat · 2018-05-25T10:44:33Z

Thanks so much for taking this on! It will make climate modelers very happy.

I think I need to defer to @spencerclark or @shoyer to answer your questions, as they understand the details better.

spencerkclark · 2018-07-11T13:24:48Z

@philippjfr @rabernat sorry I wasn't pinged here (note in the future my GitHub user name is "spencerkclark" instead of "spencerclark"), so I'm just seeing these questions now. Thanks for starting to work on this!

Which types does it make sense to support,

In principle any of the types could appear in the wild, so ideally all of them, but if we wanted to prioritize a couple and save the rest for later, the most common would probably be DatetimeGregorian and DatetimeNoLeap.

do they all support conversion to a timetuple?

Yes, all cftime types inherit from cftime.datetime which implements timetuple.

As far as I could tell this is the list of types cftime provides:
cftime._cftime.Datetime360Day
cftime._cftime.DatetimeGregorian
cftime._cftime.DatetimeJulian
cftime._cftime.DatetimeNoLeap
cftime._cftime.DatetimeProlepticGregorian

There is one more date type that cftime supports (cftime.DatetimeAllLeap), but it is rarely used.

Secondly in what context can I expect these types to be used? Will they mostly be used by xarray or should I also handle them elsewhere?

They'll probably appear the most in xarray and Iris data structures (since both support them natively), but they could show up in pure NumPy arrays (with dtype "object") or in Pandas DataFrames.

Handle matplotlib

With regard to matplotlib support, it might be worth looking at nc-time-axis. I think Iris uses it internally (e.g. here) for their built-in plotting functionality.

philippjfr · 2018-12-03T18:42:26Z

I've tried playing with nc-time-axis a bit but I can't actually get it to plot cftime objects with non-standard reference units, e.g. I haven't found any way to plot dates like this:

dates = xr.cftime_range(start='0001', periods=24, freq='MS', calendar='noleap')

My vote is to merge this without matplotlib support for now.

spencerkclark · 2018-12-03T20:02:48Z

holoviews/plotting/bokeh/util.py

+    elif pd and isinstance(date, pd.Timestamp):
+        date = date.to_pydatetime()
+    if hasattr(date, 'timetuple'):
+        dt_int = calendar.timegm(date.timetuple())*1000


Seeing this makes me a little uneasy with respect to use with cftime types. I think the built-in Python calendar module always assumes the Gregorian calendar is used; in cftime, different date types operate on different calendars, so the "seconds since 1970-01-01T00:00:00" value could be different for the same time tuple, depending on the calendar.

But perhaps there's good reason for doing things this way (I'm not familiar with how dates are handled in bokeh/holoviews)?

Thanks, definitely need another look at this. Is there a simple way to convert the values to a Gregorian calendar?

Ah, I see; this is appropriate if that is what you are looking to do.

However, I'm not sure if this is the ideal approach in general; for instance this will cause there to be extra space (and potentially a tick label) in the place of February 29th, 2000 for a no leap calendar (even though in that calendar that date does not exist). See an example in a gist with this branch. There are other possible edge cases too (in a 360-day calendar some dates do not exist in a Gregorian calendar, like February 30th).

I think nc-time-axis handles these edge cases properly, because it does not rely on converting dates to the Gregorian calendar. See how it handles the example above in this gist (I've also included an example of how to plot with dates generated from xarray.cftime_range).

I understand if it would be a lot more work to solve this in general though.

Thanks again, I'll look at nc_time_axis in a bit more detail, running your notebook is currently giving me an error though, which I had already filled here: SciTools/nc-time-axis#40.

ValueError: invalid year provided in cftime.DatetimeNoLeap(0, 11, 27, 1, 12, 0, 4, 3, 331)

What version of nc_time_axis and cftime have? I think I have the latest version of both:

print(cftime.__version__) print(nc_time_axis.__version__) 1.0.0 1.1.0

Sure thing, thanks for the link to that issue. In my notebook I was using cftime 1.0.2.1 and nc_time_axis 1.1.0. I can reproduce your issue if I downgrade cftime to version 1.0.0.

Thanks, I'll try upgrading.

Okay, I think we'll likely have to push ahead with the subtly wrong behavior in bokeh, as a proper solution would require writing custom bokeh models to correctly format dates in different calendars. Would you expect a warning when plotting dates in a non-Gregorian calendar until we've got a proper fix?

philippjfr · 2018-12-06T01:23:47Z

Here's the current status, you'll note that bokeh does not handle the no-leap case quite correctly, but as mentioned above doing so would require defining custom bokeh models.

spencerkclark

Thanks for following through with this. I really appreciate you adding support for these climate-science-specific date types!

Okay, I think we'll likely have to push ahead with the subtly wrong behavior in bokeh, as a proper solution would require writing custom bokeh models to correctly format dates in different calendars.

This is totally fine. I'm not surprised it would require a custom solution in bokeh. For now users can always switch to the matplotlib backend if they want things to be exactly correct.

Would you expect a warning when plotting dates in a non-Gregorian calendar until we've got a proper fix?

If it's not too much trouble I do think a warning would be helpful in the bokeh case. For example we now warn in xarray whenever this kind of conversion is done. At a minimum I think it should be publicly documented somewhere.

spencerkclark · 2018-12-06T11:15:42Z

holoviews/plotting/mpl/util.py

+                             'using:\n\tconda install -c conda-forge '
+                             'nc_time_axis')
+        if isinstance(value, cftime_types):
+            value = CalendarDateTime(cftime.datetime(*value.timetuple()[0:6]), value.calendar)


Is there a reason for converting to the generic cftime.datetime here and below? I think you can just pass value and value.calendar to the CalendarDateTime constructor.

Thanks, don't think there was any good reason.

spencerkclark · 2018-12-06T11:16:17Z

holoviews/plotting/bokeh/util.py

+    Attempts highest precision conversion of different datetime
+    formats to milliseconds since the epoch (1970-01-01 00:00:00).
+    If datetime is a cftime with a non-standard calendar the
+    caveats describd in cftime_to_timestamp apply.


Typo in "described"

spencerkclark · 2018-12-06T11:16:47Z

holoviews/plotting/bokeh/util.py

+    are converted to standard Gregorian calendar. This can cause
+    extra space to be added for dates that don't exist in the original
+    calendar. In order to handle these dates correctly a custom bokeh
+    model with support for other calendars would have to be defind.


Typo in "defined."

spencerkclark · 2018-12-06T11:22:49Z

holoviews/core/data/xarray.py

+        cftime._cftime.Datetime360Day,
+        cftime._cftime.DatetimeJulian,
+        cftime._cftime.DatetimeNoLeap,
+        cftime._cftime.DatetimeProlepticGregorian


I think you are missing cftime.DatetimeAllLeap here. That said, all of these are subclasses of cftime.datetime, so I think you could get away with just using cftime.datetime here, rather than enumerating all of the different subclasses (since cftime_types is only used for instance checks).

That's much easier thanks.

philippjfr · 2018-12-06T14:22:59Z

If it's not too much trouble I do think a warning would be helpful in the bokeh case. For example we now warn in xarray whenever this kind of conversion is done.

Thanks, I've adapted the error message from xarray for our purposes, e.g.:

Converting cftime.datetime from a non-standard calendar (noleap) to a standard calendar for plotting. This may lead to subtle errors in formatting dates, for accurate tick formatting switch to the matplotlib backend.

philippjfr · 2018-12-06T16:38:19Z

@jlstevens This should be ready to review and merge now.

If you're wondering about all the other changes, the dt_to_int utility was all kinds of horrendously wrong with its unit conversions, which was made up for by counter-acting it with various scaling factors in the histogram and datashading operations. The operation now actually converts between time resolutions correctly.

jlstevens · 2018-12-06T19:02:14Z

Thanks for tackling this! Tests are green so I'll merge now.

spencerkclark · 2018-12-06T20:51:13Z

Thanks again @philippjfr!

philippjfr · 2018-12-06T21:08:03Z

Thank you for being so helpful @spencerkclark!

philippjfr added the type: enhancement Minor feature or improvement to an existing feature label May 25, 2018

philippjfr force-pushed the cftime_types branch from 72d71a0 to 18d94de Compare December 3, 2018 18:27

spencerkclark reviewed Dec 3, 2018

View reviewed changes

philippjfr added 3 commits December 5, 2018 12:12

Added support for cftime types

b7d37b8

Minor fix so NdMapping allows cftimes types as keys

5ec1e7d

Add support for nc_time_axis in matplotlib backend

88bebf3

philippjfr force-pushed the cftime_types branch from 18d94de to 88bebf3 Compare December 5, 2018 13:02

philippjfr added 2 commits December 5, 2018 16:41

Fixed flakes

8b71267

Further cftime fix

eb3f0c8

philippjfr added 2 commits December 6, 2018 02:34

Improved handling of cftimes in bokeh

155cd82

Reorganized code

180117a

spencerkclark reviewed Dec 6, 2018

View reviewed changes

philippjfr added 3 commits December 6, 2018 13:33

Various datetime handling fixes

5aad6f2

Overhauled datetime handling

bef18f6

Added warning about non-standard calendars

5eee4b6

philippjfr added 3 commits December 6, 2018 14:24

Further fix

4da46b8

Added unit tests for cftime conversions

e354612

Further fixes for datetime handling

49e3b76

Fixed flakes

23fabdd

jlstevens merged commit 80dd30e into master Dec 6, 2018

philippjfr deleted the cftime_types branch December 13, 2018 04:21

spencerkclark mentioned this pull request Dec 27, 2018

holoviews / bokeh doesn't like cftime coords pydata/xarray#2164

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for cftime types #2728

Added support for cftime types #2728

philippjfr commented May 25, 2018 •

edited

rabernat commented May 25, 2018

spencerkclark commented Jul 11, 2018

philippjfr commented Dec 3, 2018

spencerkclark Dec 3, 2018

philippjfr Dec 3, 2018

spencerkclark Dec 4, 2018

philippjfr Dec 5, 2018

spencerkclark Dec 5, 2018

philippjfr Dec 5, 2018

philippjfr Dec 6, 2018 •

edited

philippjfr commented Dec 6, 2018 •

edited

spencerkclark left a comment

spencerkclark Dec 6, 2018

philippjfr Dec 6, 2018

spencerkclark Dec 6, 2018

spencerkclark Dec 6, 2018

spencerkclark Dec 6, 2018

philippjfr Dec 6, 2018

philippjfr commented Dec 6, 2018

philippjfr commented Dec 6, 2018

jlstevens commented Dec 6, 2018

spencerkclark commented Dec 6, 2018

philippjfr commented Dec 6, 2018

Added support for cftime types #2728

Added support for cftime types #2728

Conversation

philippjfr commented May 25, 2018 • edited

rabernat commented May 25, 2018

spencerkclark commented Jul 11, 2018

philippjfr commented Dec 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philippjfr Dec 6, 2018 • edited

Choose a reason for hiding this comment

philippjfr commented Dec 6, 2018 • edited

spencerkclark left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philippjfr commented Dec 6, 2018

philippjfr commented Dec 6, 2018

jlstevens commented Dec 6, 2018

spencerkclark commented Dec 6, 2018

philippjfr commented Dec 6, 2018

philippjfr commented May 25, 2018 •

edited

philippjfr Dec 6, 2018 •

edited

philippjfr commented Dec 6, 2018 •

edited