Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Pickled TimedeltaIndex from 0.19.2 cannot be unpickled in 0.20+ using pd.read_pickle #19939
Code Sample, a copy-pastable example if possible
# On pandas 0.19.2, python 2.7 import pandas as pd import cPickle as pickle td30 = pd.timedelta_range('00:30:00', '01:30:00', freq='30T') td30.to_series().to_pickle('C:/temp/td30.pkl') pd.read_pickle('c:/temp/td30.pkl') Out: 00:30:00 00:30:00 01:00:00 01:00:00 01:30:00 01:30:00 Freq: 30T, dtype: timedelta64[ns] # On latest master In : import pandas as pd In : pd.__version__ Out: '0.23.0.dev0+394.gf4c9d96' In : pd.read_pickle('C:/temp/td30.pkl') --------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) C:\projects\pandas-dk\pandas\io\pickle.py in try_read(path, encoding) 101 # We want to silencce any warnings about, e.g. moved modules. --> 102 return read_wrapper(lambda f: pkl.load(f)) 103 except Exception: C:\projects\pandas-dk\pandas\io\pickle.py in read_wrapper(func) 84 try: ---> 85 return func(f) 86 finally: C:\projects\pandas-dk\pandas\io\pickle.py in <lambda>(f) 101 # We want to silencce any warnings about, e.g. moved modules. --> 102 return read_wrapper(lambda f: pkl.load(f)) 103 except Exception: ModuleNotFoundError: No module named 'pandas.indexes.base'
Objects with TimedeltaIndexes created under pandas 0.19.2 cannot be unpickled using
Apologies if this is a dupe; while I have seen other reports of problems unpickling objects created in 0.19.2, it appears all could be fixed by using
Correct unpickling without error. Given the above example, should be the following under 0.23 master:
Pickle compat is at https://github.com/pandas-dev/pandas/blob/master/pandas/compat/pickle_compat.py
May be as simple as adding an item to
Thanks for the pointer @TomAugspurger. Looks like TimedeltaIndex was missing at 112 of pickle_compat.py. Adding it seems to fix the issue.
I'm also seeing issues with MultiIndexes with DatetimeIndex levels. I'll see if it's related and might combine into one PR.
added a commit
Mar 1, 2018
referenced this issue
Mar 1, 2018
@TomAugspurger TimedeltaIndex and Float64Index were missing from pickle_compat.py (the MultiIndex thing I saw before was actually with a Float64Index level, not the DatetimeIndex level). Adding them fixed the issue above and the other issue I found.
Happy to add a whatsnew, just wasn't sure which version this would target. Didn't add any tests because it doesn't appear any of the other index compat configs are being tested (unless I'm missing something), but I'm happy to put in a test for these if you think it's appropriate.