Delta table creation from Rust app

[johnbatty]

Hi

I'm new to Delta tables/delta-rs and am looking for some pointers/advice.

I've got a Rust app that is currently generating output files in parquet format. I want to enhance this to be able to write Delta tables. delta-rs looked like the obvious solution but looking at the examples and APIs I can't see a way to create Delta tables. I feel I must be missing something...

Questions...

• How can I create a new Delta table from a Rust app?
  • Can I do this with the delta-rs Rust APIs, or do I have to use Python?

[wjones127]

Hi @johnbatty, you should take a look at our operations module, which includes creating and writing to delta tables: https://github.com/delta-io/delta-rs/blob/main/rust/src/operations/mod.rs

Most operations are behind a feature flag datafusion-ext, since they require some form of query engine to run.

Apologies, the docs.rs is out-of-date right now, since we haven't cut a release in a while 😓. Hopefully we can do that soon.

[johnbatty]

Great - thanks for the pointer. Please could you expand on "...they require some form of query engine to run". Are you saying that I shouldn't try to use these directly? Should I be looking at doing this via datafusion?

I should make it clear that all I want to do is create a Delta table for use by other apps. I don't need to be able to query the data.

[roeap]

Right now the operations module uses datafusion. we plan to make that more pluggable in the future - so it requires some sort of query enginem which right now is datafusion :).

We did a overhaul of the operations module recently, and the create command should by now actually no longer be tied to the datafusion dependency. Other comamnds are though ...

Also this is the first iteration with a new builder pattern so things might be a bit raw. I am planning to do some homework there in the newar future though :).

[wjones127]

Are you saying that I shouldn't try to use these directly? Should I be looking at doing this via datafusion?

No, you can use the API directly. I'm just explaining that it requires that feature if you want to write data to a Delta Table.

[johnbatty]

Awesome - I'll try it out. Many thanks!