[Python] Feather format cannot store/retrieve lists correctly?

[asfimport]

I'm seeing a very weird behavior when I try to store and retrieve a Pandas data-frame using the Feather format. Simplified example:

>>> import pandas as pd
>>> df = pd.DataFrame(data={"scalar": [1, 2], "array": [[1], [7]]})
>>> df
 scalar array
0     1   [1]
1     2   [7]
>>> df.to_feather("test.ft")
>>> pd.read_feather("test.ft")
  scalar                  array
0      1                   [16]
1      2  [1045468844972122628]

As you can see, the retrieved data is incorrect. I was originally trying to use the feather-format (not using Pandas directly) and that didn't work well either.

By playing around with the data-frame that is to be stored I can also get different but still incorrect behavior, e.g. a larger list, an error that says the file size is incorrect, or simply a segmentation fault.

 

This is my first time using Feather/Arrow BTW.

Environment: Python 3.8.2
PyArrow 0.17.1
Pandas 1.0.3
Linux (Manjaro)
Reporter: Farzad Abdolhosseini

Related issues:

• [C++] IPC/Feather decompression broken for nested arrays (is duplicated by)

_{Note: This issue was originally created as ARROW-8868. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
[~farzadab] thanks for opening the issue! That's clearly buggy, and I suppose it is the same issue as the one I opened for structs (it's also a nested data type): ARROW-8860

Temporary workaround is to disable compression, as I do in the linked issue (compression="uncompressed" (the only problem is that this keyword is only passed through in pandas master, not yet released, for using it in df.to_feather)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Going to close this as a duplicate of ARROW-8860, we can continue the discussion there.