error while loading pickle as pandas DataFrame

[Code-Sage]

I ran these lines(which work fine on my laptop, for same version(0.24.2) of Pandas as on Kaggle):
pickle_filename = 'experiments.pkl'
experiments_df = pd.read_pickle(pickle_filename)

and the output was this:

ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-56-ff93e7fafa62> in <module>()
----> 1 experiments_df = pd.read_pickle(pickle_filename)

/opt/conda/lib/python3.6/site-packages/pandas/io/pickle.py in read_pickle(path, compression)
    178     try:
    179         return try_read(path)
--> 180     except Exception:  # noqa: E722
    181         if PY3:
    182             return try_read(path, encoding='latin1')

/opt/conda/lib/python3.6/site-packages/pandas/io/pickle.py in try_read(path, encoding)
    173                     lambda f: pc.load(f, encoding=encoding, compat=False))
    174             # compat pickle
--> 175             except Exception:  # noqa: E722
    176                 return read_wrapper(
    177                     lambda f: pc.load(f, encoding=encoding, compat=True))

/opt/conda/lib/python3.6/site-packages/pandas/io/pickle.py in read_wrapper(func)
    147         try:
    148             return func(f)
--> 149         finally:
    150             for _f in fh:
    151                 _f.close()

/opt/conda/lib/python3.6/site-packages/pandas/io/pickle.py in <lambda>(f)
    173                     lambda f: pc.load(f, encoding=encoding, compat=False))
    174             # compat pickle
--> 175             except Exception:  # noqa: E722
    176                 return read_wrapper(
    177                     lambda f: pc.load(f, encoding=encoding, compat=True))

/opt/conda/lib/python3.6/site-packages/pandas/compat/pickle_compat.py in load(fh, encoding, compat, is_verbose)
    210 
    211     Parameters
--> 212     ----------
    213     fh : a filelike object
    214     encoding : an optional encoding

/opt/conda/lib/python3.6/pickle.py in load(self)
   1048                     raise EOFError
   1049                 assert isinstance(key, bytes_types)
-> 1050                 dispatch[key[0]](self)
   1051         except _Stop as stopinst:
   1052             return stopinst.value

/opt/conda/lib/python3.6/pickle.py in load_stack_global(self)
   1345         if type(name) is not str or type(module) is not str:
   1346             raise UnpicklingError("STACK_GLOBAL requires str")
-> 1347         self.append(self.find_class(module, name))
   1348     dispatch[STACK_GLOBAL[0]] = load_stack_global
   1349 

/opt/conda/lib/python3.6/site-packages/pandas/compat/pickle_compat.py in find_class(self, module, name)
    133     ('pandas.tseries.tdi', 'TimedeltaIndex'):
    134         ('pandas.core.indexes.timedeltas', 'TimedeltaIndex'),
--> 135     ('pandas.indexes.numeric', 'Float64Index'):
    136         ('pandas.core.indexes.numeric', 'Float64Index'),
    137 }

/opt/conda/lib/python3.6/pickle.py in find_class(self, module, name)
   1386             elif module in _compat_pickle.IMPORT_MAPPING:
   1387                 module = _compat_pickle.IMPORT_MAPPING[module]
-> 1388         __import__(module, level=0)
   1389         if self.proto >= 4:
   1390             return _getattribute(sys.modules[module], name)[0]

ModuleNotFoundError: No module named 'pandas.core.internals.managers'; 'pandas.core.internals' is not a package

[Philmod]

Hey @Code-Sage , would you mind sharing a notebook that has the issue, so I can play with it?

[Code-Sage]

@Philmod , the notebook itself is currently incomplete, messy, and takes days to execute completely. Although, here is the whole of the erroneous part, which runs on my laptop, but neither on Kaggle nor on Microsoft Azure Notebooks.

error_loading_dataframe_pickle.ipynb.zip

[Philmod]

Doing some quick research, it may be because of pandas version.

The version 0.23.4 is installed on Kaggle.

[Code-Sage]

@Philmod I updated version(to 0.24.2) on both, my laptop and Kaggle, but still same error.

[Philmod]

1. What is the pandas version for the .pkl file? If different, can you recreate it?
2. Can you send me the updated notebook? (in the one you sent to me, pandas version was 0.23.4)

[Code-Sage]

1. The .pkl file I sent, is in 0.24.2. After thinking of the difference of versions, I re-pickled it in 0.24.2.
2. I opened the .ipynb file in a text editor, there is no pandas version specified in it. How did you check the pandas version used in the notebook? As far as I know, its related to the Python environment in which you run the notebook, and not the notebook itself.

[Code-Sage]

Plus, if you read the last line of the output i.e.

ModuleNotFoundError: No module named 'pandas.core.internals.managers'; 'pandas.core.internals' is not a package

I used the IntelliSense and actually, on Kaggle, there is no module 'pandas.core.internals.managers' , whereas it is present in my laptop's installation/environment.

[Code-Sage]

I used Pandas' msgpack option to serialize the dataframe. and the serialized dataframe worked on Pandas 0.23.4 and 0.24.2(the same serialized file that was generated on Pandas 0.24.2). That solved the issue since I could then load the dataframe in my kernel. Here is how the code looks like:

# serializing/saving dataframe to 'experiments.msgpack' file
experiments_df.to_msgpack('experiments.msgpack')

# reading/deserializing dataframe
df = pd.read_msgpack('experiments.msgpack') 

A word of caution! Pandas writes msgpack as experimental in their documentation, so, keep an eye out for issues caused by it.

[lazyoracle]

@Philmod This issue is still persisting. The version of pandas in Kaggle is 0.23.4 while my pickle is saved using 0.24.1. There is a known bug is opening pickle files saved using newer versions of pandas as discussed here.
So when I try to get a newer version of pandas installed in my Kaggle notebook kernel, it actually doesn't recognise the newer version and continues to load the erstwhile 0.23.4. Kernel and sample datasets are listed below.
Pandas Cross-Version Pickle Loading Error
test-data

[Code-Sage]

@Philmod This issue is still persisting. The version of pandas in Kaggle is 0.23.4 while my pickle is saved using 0.24.1. There is a known bug is opening pickle files saved using newer versions of pandas as discussed here.
So when I try to get a newer version of pandas installed in my Kaggle notebook kernel, it actually doesn't recognise the newer version and continues to load the erstwhile 0.23.4. Kernel and sample datasets are listed below.
Pandas Cross-Version Pickle Loading Error
test-data

@lazyoracle , you may try the dataframe.to_msgpack(...) option if that's okay for your job. It works across multiple versions of Pandas, I used it for 0.23.4 and 0.24.1(I think this was the latest back then). Though keep in mind this is still an experimental library/functionality.

[lazyoracle]

@Code-Sage Thanks for the suggestion but I do not want to use the msgpack() option since it's an experimental library and my data files being the size of 3 GiB, as outputs from experimental runs, I can not afford to have them corrupted. Hence, tagging @Philmod to figure out if there is any suggestion on why even after installing pandas==0.24.1, the Kaggle kernel shows the version to be 0.23.4. That is a more broad issue that I am interested in understanding and resolving.