GDS Projection from polars / pandas dataframe or arrow table

[Mintactus]

The new Polars dataframe multi engine is absolutely a must in the data industry.
After using it for months, the performance benifits are insane, adios pandas, your time has come.

Allowing GDS to export as and create projections from polars dataframes would be natural today. ( At least once Polars will be out of Alpha )

Even better, GDS being based on apache arrow, I think it would make sens for GDS to create projection directly from an arrow table ? This will makes it agnostic to the engine processing the data.

[Mats-SX]

Duplicate of #653

[Mats-SX]

While projection and export are two distinct feature in GDS and its representation in the GDS Python Client, the question of what kind of DataFrame libraries they accept is seen as a global integration. If we added support for Polars, it should apply for both export and projections.

For now, the same workaround to convert to/from pandas data frames will assist workflows based on Polars. My discussion in the other issue applies similarly for projection, where we make use of Table.from_pandas() in pyarrow, but there is no Table.from_polars().

[Mintactus]

Since, polars and pandas ( at least the recent version ) and GDS, and so much more on the market all shared one thing in commun, apache arrow format, would it be a solution to simply import and export ( optionally or as a default behavior ) in arrow table format ?

This will makes GDS agnostic the to engine processing the data before they are shipped into or out of GDS?

Thanks

[Mats-SX]

It is a possibility. But the pyarrow.Table type is not as ubiquitous as the pandas.DataFrame type. It is nice to have DataFrame in the API.

But we can have a polymorphic parameter set, and allow passing in pyarrow.Table objects directly. It would be some work to accomplish, but it would be possible.

[Mintactus]

At the moment it's probably not critical but over time if there are multiple engines on the market using apache arrow it could be a quicker way to make GDS agnostic to these engines. Polars / Pandas are for now the main ones I know, polars seriously kicking the ass of pandas.

…

On Tue, Jun 11, 2024 at 6:27 AM Mats Rydberg ***@***.***> wrote: It is a possibility. But the pyarrow.Table type is not as ubiquitous as the pandas.DataFrame type. It is nice to have DataFrame in the API. But we can have a polymorphic parameter set, and allow passing in pyarrow.Table objects directly. It would be some work to accomplish, but it would be possible. — Reply to this email directly, view it on GitHub <#654 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHIBVDPLHBDBZWDK4IAHPXLZG3GJBAVCNFSM6AAAAABIWRQEOWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRQGM4DQNZRGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>