Bug in UTCDateTime

[jsaul]

Python 2.7.6 (default, Mar 22 2014, 22:59:38) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from obspy.core import UTCDateTime
>>> t = UTCDateTime("2014-05-23T22:35:30")
>>> print t
2014-05-23T22:35:30.000000Z
>>> t = UTCDateTime("2599-05-23T22:35:30")
>>> print t
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/obspy/core/utcdatetime.py", line 895, in __str__
    return "%s%sZ" % (self.strftime('%Y-%m-%dT%H:%M:%S'),
  File "/usr/lib/python2.7/dist-packages/obspy/core/utcdatetime.py", line 1126, in strftime
    return self._getDateTime().strftime(format)
  File "/usr/lib/python2.7/dist-packages/obspy/core/utcdatetime.py", line 492, in _getDateTime
    return datetime.datetime.utcfromtimestamp(self.timestamp)
ValueError: timestamp out of range for platform time_t

This looks like a bug to me. I found this when working with FDSN StationXML produced by IRIS, where end times (e.g. of a sensor operation period) in the future are often expressed using times very far in the future (commonly end of year 2599), so this is not an esoteric example.

ObsPy version is 0.9.2

[markcwill]

I can't reproduce, @jsaul are you running a 32-bit or embedded platform? Looks like the problem is your version of python has a datetime module that is storing time in a time_t integer that doesn't go that high.
http://en.wikipedia.org/wiki/Year_2038_problem

(obspy-dev)chase:~/Code/python/obspy % uname -srvpo                                          [git|master]
Linux 3.11.0-15-generic #25-Ubuntu SMP Thu Jan 30 17:22:01 UTC 2014 x86_64 GNU/Linux

(obspy-dev)chase:~/Code/python/obspy % python                         [git|0.9.2]
Python 2.7.5+ (default, Feb 27 2014, 19:37:08) 
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from obspy.core import UTCDateTime
>>> t = UTCDateTime("2599-05-23T22:35:30")
>>> print t
2599-05-23T22:35:30.000000Z
>>> t.timestamp
19861713330.0
>>> 

[jsaul]

Good point, I can reproduce that the error doesn't occur on a 64-bit platform, so it's clearly as you say the Year 2038 problem. The timestamp is correct on the 32-bit platform, so it's mostly a formatting issue.

In my client code it can be handled by something like

if cha.end_date.timestamp > 1<<31:
    # workaround

but it would of course be nice to have a generic solution. I'll have a closer look at this.

[krischer]

We've already seen some manifestations of this issue in the past. It happens on some (not all if I am not mistaken) 32 bit platforms as already noted. UTCDateTime internally uses a double precision float to store the current time as a timestamp and the problem occurs at the conversion back to datetime.datetime:

https://github.com/obspy/obspy/blob/master/obspy/core/utcdatetime.py#L505-506

This is used every time a string representation or single components of a datetime like the year, month, ... are requested.

It might be possible to change the offending line and use datetime.timedelta instead and just add it to timestamp 0? Alternatively this will probably work: http://stackoverflow.com/a/12458703
We might have to be careful with some timezone issues but the UTCDateTime test suite it fairly comprehensive.

If performance is an issue we can use try/except and fall back to the slower solution if needed.

@jsaul: Can you test these solutions? None of my installations has this issue.

[jsaul]

Python 2.7.6 (default, Mar 22 2014, 22:59:38) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>> datetime.datetime.utcfromtimestamp(1<<30)
datetime.datetime(2004, 1, 10, 13, 37, 4)
>>> datetime.datetime.utcfromtimestamp(1<<31)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: timestamp out of range for platform time_t
>>> datetime.datetime.utcfromtimestamp(0) + datetime.timedelta(seconds=1<<30)
datetime.datetime(2004, 1, 10, 13, 37, 4)
>>> datetime.datetime.utcfromtimestamp(0) + datetime.timedelta(seconds=1<<31)
datetime.datetime(2038, 1, 19, 3, 14, 8)
>>> datetime.datetime.utcfromtimestamp(0) + datetime.timedelta(seconds=1<<32)
datetime.datetime(2106, 2, 7, 6, 28, 16)
>>> 

Not the prettiest workaround, but it does work on 32-bit now, thanks!

[barsch]

we actually did already use the datetime.timedelta trick to handle precision issues - so using the code above does actually speed up things ...

fixed in master - closing

[megies]

Thanks for fixing @barsch. @barsch, @krischer, this is a huge issue that is practically guaranteed to render the whole obspy.station part unusable when working on larger datasets, so I backported this to the releases branch and I would even suggest to make a 0.9.3 bugfix release for it soon.