Support pyarrow LargeListType

[Modexus]

Fixes #6834

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Modexus]

Fixed the conversion from pyarrow to python Sequence features.

There is still an issue that if features are passed the Sequence always forces conversion to ListArray.
This probably causes issues if the LargeListArray is actually needed.

There doesn't seem to be a great solution since this list is created solely on the schema for Sequence.
One solution would be to always use LargeListArray instead.

[albertvillanova]

Hi, @Modexus.

Sorry for the late review.

In relation with the conversion of Sequence, we had a discussion time ago, and we proposed to add a parameter to Sequence to specify to convert it to whether regular list or large list. See comments:

• support LargeListArray in pyarrow #4800 (comment)
• support LargeListArray in pyarrow #4800 (comment)

Would you be interested in implementing this in your PR?

[albertvillanova]

I am retaking this PR because we would like to have this feature implemented.