Pre-epoch dates giving OverflowError in mktime with HDFStore

[salestial]

I am having trouble with pre-epoch dates working with HDFStore, getting an OverflowError in mktime, as shown below.

Sample code is:

    import numpy as np
    from pandas import *


    def panda_test():

        # generate some data
        data = np.random.rand(50,5)    
        # generate some dates
        dates = DateRange('1/1/1969',periods=50)    
        # generate column headings
        cols = ['A','B','C','D','E']

        df = DataFrame(data,index=dates,columns=cols)

        # save to HDF5Store
        store = HDFStore('bugzilla.h5', mode='w')
        store['df'] = df # This gives: OverflowError: mktime argument out of range
        store.close()


    if __name__ == '__main__':
        panda_test()

The error I get is:

    Traceback (most recent call last):
      File "C:\Users\Salman Ansari\Documents\Python\VAR\src\bugzilla.py", line 27, in <module>
        panda_test()
      File "C:\Users\Salman Ansari\Documents\Python\VAR\src\bugzilla.py", line 22, in panda_test
        store['df'] = df
      File "C:\Python27\lib\site-packages\pandas-0.4.0-py2.7-win32.egg\pandas\io\pytables.py", line 122, in __setitem__
        self.put(key, value)
      File "C:\Python27\lib\site-packages\pandas-0.4.0-py2.7-win32.egg\pandas\io\pytables.py", line 260, in put
        comp=compression)
      File "C:\Python27\lib\site-packages\pandas-0.4.0-py2.7-win32.egg\pandas\io\pytables.py", line 327, in _write_to_group
        wrapper(value)
      File "C:\Python27\lib\site-packages\pandas-0.4.0-py2.7-win32.egg\pandas\io\pytables.py", line 325, in <lambda>
        wrapper = lambda value: handler(group, value)
      File "C:\Python27\lib\site-packages\pandas-0.4.0-py2.7-win32.egg\pandas\io\pytables.py", line 335, in _write_frame
        self._write_block_manager(group, df._data)
      File "C:\Python27\lib\site-packages\pandas-0.4.0-py2.7-win32.egg\pandas\io\pytables.py", line 346, in _write_block_manager
        self._write_index(group, 'axis%d' % i, ax)
      File "C:\Python27\lib\site-packages\pandas-0.4.0-py2.7-win32.egg\pandas\io\pytables.py", line 432, in _write_index
        converted, kind, _ = _convert_index(index)
      File "C:\Python27\lib\site-packages\pandas-0.4.0-py2.7-win32.egg\pandas\io\pytables.py", line 698, in _convert_index
        for v in values], dtype=np.int64)
    OverflowError: mktime argument out of range
    Closing remaining open files: bugzilla.h5... done

Any ideas, please?

[wesm]

This is caused by a bug in the Python interpreter. If I'm reading the Python bug tracker correctly it was fixed in Python 2.7.2:

http://bugs.python.org/issue1726687

what version of Python are you using?

[wesm]

Perhaps I should put a warning letting people know to upgrade their Python version if they get this bug?

[salestial]

I am using Python 2.7.2 on Windows 7 (C:\Python27\python.exe 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)]) so I am not sure this resolves it?

[wesm]

According to the Python docs this is a platform specific problem and I guess does not have a short-term resolution

http://docs.python.org/library/time.html#time.mktime

The best immediate solution I have for you is: get 64-bit Python which does not have this problem. Longer term I am going to integrate the NumPy datetime64 dtype into pandas which will solve this. But that's only something I can see happening within the next few months.

I'm going to leave the issue open so I don't forget about it.

[salestial]

Thanks and look forward to it.

[wesm]

Alternative requiring NumPy > 1.6, I suggested on the mailing list

int_index = np.array(date_index, dtype='M8[us]').view(np.int64)

then on the way back:

date_index = int_index.view('M8[us]').astype('O')

[wesm]

This is resolved in git master by the move to datetime64