Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Status of Writing to Iceberg? #2

Closed
jacksonrnewhouse opened this issue Mar 21, 2023 · 6 comments
Closed

Status of Writing to Iceberg? #2

jacksonrnewhouse opened this issue Mar 21, 2023 · 6 comments

Comments

@jacksonrnewhouse
Copy link

Hi there, I'm building something that uses datafusion for parts of the planning and would like to be able to write results directly to Iceberg. Is the intent of this project to support that eventually? How much effort would it take and could I potentially help with that?

@JanKaul
Copy link
Owner

JanKaul commented Mar 22, 2023

Hi, it's definitely planned to support writing to iceberg. I just didn't get around to implementing it.
And I haven't figured out a perfect way to do so. I think Datafusion currently doesn't have the functionality for INSERT INTO operations. So the current solution probably requires some external writer.
We can have a look at how delta_rs is doing it.

@jacksonrnewhouse
Copy link
Author

I'm currently only using the parsing and planning components of DataFusion and performing the execution separately. Since 17.0 the DataFusion grammar now supports WriteOps, via this PR: apache/datafusion#4902

@JanKaul
Copy link
Owner

JanKaul commented Mar 22, 2023

Okay great, I will have a look. If you have an idea how this could be implemented it would be great if you could share it here. I will also report my findings.

@doki23
Copy link

doki23 commented Mar 31, 2023

Do we have rust api for writing?

@JanKaul
Copy link
Owner

JanKaul commented Mar 31, 2023

No, not yet. But I will look into it in the coming days.

@JanKaul
Copy link
Owner

JanKaul commented Jan 20, 2024

It's been a while but the crate now supports writing to iceberg tables with datafusion.

@JanKaul JanKaul closed this as completed Jan 21, 2024
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants