Add support for FixedSizeList from Arrow instead of crashing on from_arrow

[marsupialtail]

Polars version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Issue description

Arrow has a data type called FixedSizeList. Currently making Polars DataFrame from an Arrow table with this doesn't work.

This data type is quite important for embeddings etc.

Reproducible example

First you need to download the parquet dataset here: https://drive.google.com/file/d/1sk9SM589YTjPFYI1RsxcdJf5WmmZJTzr/view?usp=sharing

Then just try to read the parquet file polars.read_parquet

Interestingly converting first to Pandas and reading from Pandas works with Polars.

Expected behavior

Works.

Installed versions

0.16.15

[marsupialtail]

@ritchie46 I would like to contribute a fix to this. Can you point me to some files I should look at?

[kylebarron]

This is a duplicate of #4014 I think

[kylebarron]

I've been meaning to add FixedSizeList support for a while but haven't found the time yet. The previous suggestion from Ritchie for reference is here: https://discord.com/channels/908022250106667068/908022461751234570/1074262804183404574

Yes, add a fixed list type (feature gated) and then implement the Series trait. Should be pretty doable.

This diff: #6374 and this diff: #6679

[kylebarron]

This issue can probably be closed given #8943

[stinodego]

Yep, this now works fine:

import pyarrow as pa

import polars as pl

values = pa.array([1, 2, 3, 4])
arr = pa.FixedSizeListArray.from_arrays(values, 2)
s = pl.from_arrow(arr)
print(s)

shape: (2,)
Series: '' [array[i64, 2]]
[
        [1, 2]
        [3, 4]
]

[claysmyth]

Got this error after using df.write_parquet()

PanicException: not yet implemented: Writing FixedSizeList to parquet not yet implemented, is there a work-around?

[stinodego]

Got this error after using df.write_parquet()

PanicException: not yet implemented: Writing FixedSizeList to parquet not yet implemented, is there a work-around?

Could you please open a new issue for this?

[franz101]

#14876