BUG: TypeError: _reconstruct: First argument must be a sub-type of ndarray

[JamesAllingham]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample

import pandas as pd
pd.read_pickle("filtered_data.pickle")

Problem description

I am able to import the dataframe (which can be downloaded in zipped form from here) with Pandas version 0.24.2 but get the following error with version 1.0.3:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/homes/jua23/.virtualenvs/arch_uncert/lib/python3.7/site-packages/pandas/io/pickle.py", line 182, in read_pickle
    return pickle.load(f)
TypeError: _reconstruct: First argument must be a sub-type of ndarray

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.7.final.0
python-bits : 64
OS : Linux
OS-release : 5.3.0-42-generic
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 1.0.3
numpy : 1.18.2
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 46.0.0
Cython : 0.29.15
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.1
IPython : 7.13.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.2.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[TomAugspurger]

What are the dtypes on the DataFrame?

And if you're able to, can you provide code to create the DataFrame? I'm guessing no maintainers will try to load your pickle file since it's a potential security issue.

[JamesAllingham]

Unfortunately I am not the person who created the DataFrame and I have no way of getting the code which created it. Here is what the head looks like:

   Year  Month  DayofMonth  DayOfWeek  DepTime  ArrTime  ArrDelay  AirTime  Distance  plane_age
0  2008      1           3          4   1343.0   1451.0      16.0     55.0       393        2.0
1  2008      1           3          4   1758.0   1854.0      -6.0     45.0       303        2.0
2  2008      1           3          4   1516.0   1614.0       9.0     49.0       296        2.0
3  2008      1           3          4   1921.0   2017.0     -18.0     46.0       321        2.0
4  2008      1           3          4    617.0    743.0       3.0     68.0       446        2.0

And here are the data types (df.dtypes):

Year            int64
Month           int64
DayofMonth      int64
DayOfWeek       int64
DepTime       float64
ArrTime       float64
ArrDelay      float64
AirTime       float64
Distance        int64
plane_age     float64
dtype: object

Thanks for the help!

[TomAugspurger]

If you save that DataFrame with pandas 0.24.2 and load it with 1.0.3, do you see the exception?

[JamesAllingham]

No, it works fine in that case.

[jreback]

you pickle is way too old
that error was for when series inherited from ndarrays

1.x doesn’t suooort pickles that old

[TomAugspurger]

Ah, I misread it as your pickle was created in 0.24.2. Yes, we definitely don't support reading pickle files from that old of a version.

[JamesAllingham]

I think it would be great to have an informative error message that explains the problem. Or better yet, backwards compatibility for these old pickles. I imagine that there are plenty of old pickles floating around that people will want to be able to import into the new versions of Pandas. Of course, I realize that this might be more effort than it is worth to implement and maintain, so the error message might be a better solution.

Thanks for the help! :)

[lawrennd]

James, do you have the updated pickle? If so then I can fix this at source (I've just been discovering the bug when I try and access it!).

[JamesAllingham]

Hey Neil! Yes, here you go: https://www.dropbox.com/s/sxf3z1g7ltr60dy/filtered_data2.pickle?dl=0

Let me know If that doesn't work.

By the way, what is the source for this file? I acquired it through a long chain of people. It would be good to know where the official source is!