Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add delta lake file support #752

Open
the-destro opened this issue Dec 1, 2023 · 4 comments
Open

Add delta lake file support #752

the-destro opened this issue Dec 1, 2023 · 4 comments
Labels
kind:feature New feature or request

Comments

@the-destro
Copy link

Python Polars has support for delta lake files already though I can't find the function exposed in the rust package.

@josevalim
Copy link
Member

Can you please link to the Python version of the function?

@watsy0007
Copy link

watsy0007 commented Dec 24, 2023

@the-destro
https://pola-rs.github.io/polars/py-polars/html/reference/io.html#delta-lake
Is scan_delta, read_delta and DataFrame.write_delta() these 3 functions ?

https://github.com/pola-rs/polars/blob/40d3e0818408d836abf6c31146a3f69fd628f0fb/py-polars/polars/io/delta.py#L295

Make sure to install deltalake>=0.8.0. Read the documentation
here <https://delta-io.github.io/delta-rs/python/installation.html>_.

The rust package repository is https://github.com/delta-io/delta-rs

@josevalim
Copy link
Member

Btw, isn't delta-lake storage pretty much Parquet files? Could you access them directly instead? Writing them would be a bit more complicated though.

@josevalim josevalim changed the title [Feature Request] Add delta lake file support Add delta lake file support Apr 16, 2024
@josevalim josevalim added the kind:feature New feature or request label Apr 16, 2024
@watsy0007
Copy link

Btw, isn't delta-lake storage pretty much Parquet files? Could you access them directly instead? Writing them would be a bit more complicated though.

Yes, that's correct. In our company, we integrate DuckDB, dbt, and Delta Lake with Python for business operations. I'm currently considering replacing some of these components with Elixir.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants