Timezones and Datetime64

[PythonCHB]

A new version of datetime64 dtype was added to numpy recently, and as of 1.7, is considered experimental.

As of 1.7, datetime64 attempts to handle timezones by:

• Assuming all datetime64 objects are in UTC
• Applying timezone offsets when parsing ISO 8601 strings
• Applying the Locale timezone offset when the ISO string does not specify a TZ.
• Applying the Locale timezone offset when printing, etc.

(note that no timezone handling is done when converting to/from the stdlib datetime object)

However, this behavior makes it very difficult to use datetime64 in a timezone agnostic way, or if you need to work with a timezone that is not the system Locale one.

Complete and proper timezone management could be a very useful, but the current system is only useful in a few small cases, and makes common use cases very difficult. Good timezone handling is a large and difficult problem -- it is unclear whether numpy wants to take it on.

There has been much discussion on the numpy mailing list, and a new datetime64 NEP is progress, but in the meantime, I propose we make the following changes to make datetime64 far more useful and less prone to error:

• Assume that datetime64 is timezone "naive", as the standard library datetime object does -- it is up to the user to do any time zone offset conversions.
• Do not apply a timezone offset when converting to an ISO string
• Never use the Locale settings for anything, without the user explicitly asking for it.
• raise an exception when parsing an ISO 8601 string that has a timezone offset other than 0 (Z)

[njsmith]

If we're naive, shouldn't we also refuse to process a timezone offset of 0,
and only accept ISO 8601 strings that leave the offset unspecified?

"Naive" and "UTC" are different timezones... note that these return
different values:
dateutil.parser.parse("2010-01-01T00:00:00")
dateutil.parser.parse("2010-01-01T00:00:00Z")

On Mon, Apr 29, 2013 at 5:54 PM, Christopher H.Barker, PhD <
notifications@github.com> wrote:

A new version of datetime64 dtype was added to numpy recently, and as of
1.7, is considered experimental.

As of 1.7, datetime64 attempts to handle timezones by:

• Assuming all datetime64 objects are in UTC
• Applying timezone offsets when parsing ISO 8601 strings
• Applying the Locale timezone offset when the ISO string does not
  specify a TZ.
• Applying the Locale timezone offset when printing, etc.

(note that no timezone handling is done when converting to/from the stdlib
datetime object)

However, this behavior makes it very difficult to use datetime64 in a
timezone agnostic way, or if you need to work with a timezone that is not
the system Locale one.

Complete and proper timezone management could be a very useful, but the
current system is only useful in a few small cases, and makes common use
cases very difficult. Good timezone handling is a large and difficult
problem -- it is unclear whether numpy wants to take it on.

There has been much discussion on the numpy mailing list, and a new
datetime64 NEP is progress, but in the meantime, I propose we make the
following changes to make datetime64 far more useful and less prone to
error:

• Assume that datetime64 is timezone "naive", as the standard library
  datetime object does -- it is up to the user to do any time zone offset
  conversions.
• Do not apply a timezone offset when converting to an ISO string
• Never use the Locale settings for anything, without the user
  explicitly asking for it.
• raise an exception when parsing an ISO 8601 string that has a
  timezone offset other than 0 (Z)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3290
.

[PythonCHB]

Well, dateutil returns a time-zone aware datetime object when you give it a time zone -- so it CAN return different values with and without a TZ offset.

I just figured it wouldn't change the value, so why not. But sure, perhaps it would be less likely to cause errors later on to not except ANY timezone offset in a ISO string.

[njsmith]

I mean that it is a different value precisely in the sense that time-zone
aware datetimes and time-zone naive datetime objects are very different,
incommensurate things, even if they have some of the same numbers stashed
in them:

In [6]: dateutil.parser.parse("2010-01-01T00:00:00")
Out[6]: datetime.datetime(2010, 1, 1, 0, 0)

In [7]: dateutil.parser.parse("2010-01-01T00:00:00Z")
Out[7]: datetime.datetime(2010, 1, 1, 0, 0, tzinfo=tzutc())

In [8]: dateutil.parser.parse("2010-01-01T00:00:00") -
dateutil.parser.parse("2010-01-01T00:00:00Z")
TypeError: can't subtract offset-naive and offset-aware datetimes

Saying "oh well they're not that different it might be convenient and who
will really notice anyway" is exactly the kind of hand-wavy reasoning that
got us into this mess :-).

On Tue, Apr 30, 2013 at 12:33 AM, Christopher H.Barker, PhD <
notifications@github.com> wrote:

Well, dateutil returns a time-zone aware datetime object when you give it
a time zone -- so it CAN return different values with and without a TZ
offset.

I just figured it wouldn't change the value, so why not. But sure, perhaps
it would be less likely to cause errors later on to not except ANY timezone
offset in a ISO string.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3290#issuecomment-17209152
.

[njsmith]

@wesm: Do you have any opinion on the proposed changes? I'm extremely
reluctant to mess with this stuff further without pandas signing off.

On Tue, Apr 30, 2013 at 8:18 AM, Nathaniel Smith njs@pobox.com wrote:

I mean that it is a different value precisely in the sense that time-zone
aware datetimes and time-zone naive datetime objects are very different,
incommensurate things, even if they have some of the same numbers stashed
in them:

In [6]: dateutil.parser.parse("2010-01-01T00:00:00")
Out[6]: datetime.datetime(2010, 1, 1, 0, 0)

In [7]: dateutil.parser.parse("2010-01-01T00:00:00Z")
Out[7]: datetime.datetime(2010, 1, 1, 0, 0, tzinfo=tzutc())

In [8]: dateutil.parser.parse("2010-01-01T00:00:00") -
dateutil.parser.parse("2010-01-01T00:00:00Z")
TypeError: can't subtract offset-naive and offset-aware datetimes

Saying "oh well they're not that different it might be convenient and
who will really notice anyway" is exactly the kind of hand-wavy reasoning
that got us into this mess :-).

On Tue, Apr 30, 2013 at 12:33 AM, Christopher H.Barker, PhD <
notifications@github.com> wrote:

Well, dateutil returns a time-zone aware datetime object when you give it
a time zone -- so it CAN return different values with and without a TZ
offset.

I just figured it wouldn't change the value, so why not. But sure,
perhaps it would be less likely to cause errors later on to not except ANY
timezone offset in a ISO string.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3290#issuecomment-17209152
.

[PythonCHB]

@njsmith ,

Yes. Naive and Tz-aware date times are quite different. But doing tz-awareness right is really hard. Even if we can Agree on what we want it to do. This my proposal to stick with naive. Which, indeed it currently is--the current code simply uses some unfortunate defaults. I'd like to see that change to something far more useful.

If we go with naive, then really the only open question is when to raise errors when parsing ISO strings. If numpy only supports naive date times, then I see little chance of confusion if it accepts a tzoffset of 0 -- but consistancy is good, so no- offset-allowed is good too.

[wesm]

I think that datetime64 should always be implicitly a UTC timestamp, and can be interpreted as being in some time zone as the user desires. That's the approach we're taking in pandas right now, e.g.:

In [2]: stamp = pd.Timestamp('2013-05-03T23:00-0500')

In [3]: stamp
Out[3]: <Timestamp: 2013-05-03 23:00:00-0500 , tz=tzoffset(None, -18000)>

In [4]: stamp.u
stamp.utcfromtimestamp  stamp.utcoffset         
stamp.utcnow            stamp.utctimetuple      

In [4]: stamp.value
Out[4]: 1367640000000000000

In [5]: stamp.tz_convert('utc')
Out[5]: <Timestamp: 2013-05-04 04:00:00+0000 UTC, tz=UTC>

In [6]: stamp.tz_convert('utc').value
Out[6]: 1367640000000000000

The Timestamp object is simply a user-friendly "box" and subclass of datetime.datetime that holds a UTC datetime64 timestamp (in nanoseconds).

[wesm]

I should also add that unfortunately we had to kind of do everything ourselves in pandas because the NumPy datetime64 stuff has been in flux (I wrote earlier "mess" but that was putting it a bit strongly, sorry). I also disagreed with some of the decisions that Mark made in the API. I think interpreting a timestamp as naive and not localtime is a good idea, I didn't like the "convert this naive string to UTC given local timezone" which is how I recall numpy 1.7 working

[charris]

@mwiebe Just a heads up to the discussion on the datetime issue.

[PythonCHB]

Panda's proves the point: a naive datetime may not be as useful as a full-featured Tz-aware one but it is very useful, and users can patch their own TZ handling on top of it. Half-baked TZ support is next to useless. I'm using numpy1.7 datetime64, but frankly, I don't know that I'm buying anything over simply using a regular array that I interpret as seconds-since-the-epoc.

Maybe we should borrow Panda's code?

As for Naive vs. UTC -- they are almost exactly the same -- the only difference may be how you handle an ISO string as input: add/subtract the offset, or don't allow an offset. As long as you don't apply the locale offset without the user specifically asking for it, I'm good either way, though I think calling it "UTC" buys very little over naive, but does add a tiny bit of extra room for error.

So: I propose either Naive or UTC--implementer's choice.

-Chris

[notbanker]

Wes I am curious about the motivation for allowing pd.to_datetime( ) to apparently select a timezone from the local system when supplied only a long. That is:

stamp = pd.Timestamp('2013-05-03T23:00')

pd.to_datetime( stamp, box = False )
Timestamp('2013-05-03 23:00:00', tz=None)

versus

pd.to_datetime( stamp.value, box = False )
numpy.datetime64('2013-05-03T19:00:00.000000000-0400')

Took me by surprise, but I'm sure you've thought it through.

[charris]

This is still under discussion/development for 1.9

[PythonCHB]

On Sat, Feb 22, 2014 at 2:46 PM, Charles Harris notifications@github.comwrote:

This is still under discussion/development for 1.9

great -- any kind of timeframe on that? I think we have a little more
consensus building to to, but really the key blocker is someone that has
the time and skills to sit down and do the work. It wuold be nice not to
leave it ot the last minute like right before 1.8...

-Chris

Christopher Barker, PhD

Python Language Consulting

• Teaching
• Scientific Software Development
• Desktop GUI and Web Development
• wxPython, numpy, scipy, Cython