Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for read_delta #14

Open
MrPowers opened this issue Jul 20, 2023 · 2 comments
Open

Add support for read_delta #14

MrPowers opened this issue Jul 20, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@MrPowers
Copy link

Problem description

The Polars CLI supports read_csv and read_parquet, but it doesn't look like read_delta is supported yet.

Polars supports read_delta, so I am hoping it won't be too hard to add it to the Polars CLI.

Screenshot 2023-07-20 at 10 37 17 AM
@MrPowers MrPowers added the enhancement New feature or request label Jul 20, 2023
@stinodego
Copy link
Member

read_delta is implemented in Python, not on the Rust side. So the CLI does not have access to it.

I would be in favor of adding this, but it's not an easy one. I believe @ritchie46 mentioned at one point that he preferred to have the read_delta functionality on the Python side only, but I'm not really sure why (maybe I am misremembering).

The dependency we use for read_delta in Python is implemented in Rust, so it's probably possible to have this on the Rust side.

@ritchie46
Copy link
Member

ritchie46 commented Jul 20, 2023

Delta-rs uses arrow-rs, so we would need to compile a whole new arrow implementation. We could add it under a separate crate that users could draw in if they want to pay the compilation cost.

For the CLI it can be fine if we have binaries.

@universalmind303 universalmind303 transferred this issue from pola-rs/polars Sep 8, 2023
@stinodego stinodego changed the title Add read_delta to Polars CLI Add support for read_delta Oct 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Ready
Development

No branches or pull requests

3 participants