ADBC Driver

[wjones127]

Description

I'm thinking about something we might want to work toward in 2023, and an ADBC driver might be a good focus. We would aim to use this to support integration with DuckDB and Polars, as well as to upgrade the Python reader and writer. (DataFusion could also use this, but likely we will have direct support using the underlying implementation; see the doc for details.)

Draft design doc here, open to comments: https://docs.google.com/document/d/1ud-iBPg8VVz2N3HxySz9qbrffw6a9I7TiGZJ2MBs7ZE/edit?usp=sharing

Use Case

Make it easy to build connectors to Delta-rs, no matter the language, and with strong guarantees of implementation correctness.

Related Issue(s)

Work on ADBC driver (in rough priority order):

• Create initial ADBC connector that provides access to delta lake catalogue
• Add support for SELECT queries
• Set up DAT reader tests for connector
• Set up object store parameters and integration tests
• Add support for SELECT COUNT(*) queries
• Add support for OPTIMIZE and VACUUM
• Add support for CREATE and INSERT INTO/OVERWRITE
• Set up DAT writer tests
• Add support for UPDATE
• Add support for DELETE
• Add support for MERGE
• Add Substrait support for all queries

Downstream work enabled by driver:

• Switch Python bindings to read and write from ADBC driver
• Create R bindings
• Create DuckDB plugin
• Create Polars integration
• Ibis integration?

[DOH-WLD0303]

Has there been any progress towards this? I would be very interested in seeing what I could do to contribute here!

[wjones127]

Hi @DOH-WLD0303, nice to hear!

I'm currently working upstream on the foundations on apache/arrow-rs#3582

Once, that's done, I'll finish up my work in #945. And then once that is done more of these issues will be available. It may take me a few weeks to finish those though.

In the meantime, if you want you would work in parallel on implementing or improving some of the underlying operations in https://github.com/delta-io/delta-rs/tree/main/rust/src/operations, that would be helpful as well. I think OPTIMIZE, DELETE, and UPDATE are ones we still haven't done.

[DOH-WLD0303]

@wjones127 , sincerest apologies but my work timeline isn't going to give me the bandwidth to assist on this for right now. I've never dipped my toes into rust either which is definitely a barrier for me unfortunately. I'm hoping I can learn enough at some point to contribute a little!