Support reading delta tables #190
Conversation
| @@ -27,6 +27,7 @@ thiserror = "1" | |||
| smartstring = { version = "1" } | |||
| serde_json = { version = "1" } | |||
| either = "1.9" | |||
| deltalake = "0.17.1" | |||
There was a problem hiding this comment.
I'm a bit hesitant to add this in as a dependency.
this pulls in the entire arrow-rs project as well as many other heavy dependencies.
There was a problem hiding this comment.
It makes sense, I will keep looking for alternatives. Thx
There was a problem hiding this comment.
FWIW, I think the "ideal" approach would be first to upstream some work to delta-rs and expose a javascript library. Then we could use the JS library directly like py-polars does, and do a bunch of the logic on the JS side using apache-arrow as an optional dependency.
If the apache-arrow npm package becomes a point of contention, we could likely bypass it by dynamically linking to the delta js bindings directly.
There was a problem hiding this comment.
delta-rs team is busy with C# implementation now. :)
There was a problem hiding this comment.
There was effort to use delta-sharing using JS and Danfo.js.
Adding support for reading delta tables thru Rust deltalake
To close #176