Rust Support

[D1plo1d]

DuckDB looks like it is solving many issues with embedded databases for me but I work in Rust (and have little C++ experience). Are there plans to release a Rust crate around duckdb?

My ideal solution would be for DuckDB support to be added to the sqlx crate.

[hannes]

We have little experience in Rust, but welcome contributions of course. Indeed a possibility is to add to sqlx.

[blackrez]

I'm interested too but I see that rust can't call c++. Does duckdb have a C interface ?

[Mytherin]

Yes, the C interface is here.

[blackrez]

Nice, so it should be possible to create a rust binding. I will give a try this next days. Do you have a slack/gitter/IRC channel for my future stupid questions ?

[Mytherin]

We do not have these channels, but feel free to ask any questions you have in the issue here. That way other people can also see the discussions (and potentially respond).

[wangfenjin]

Hi, I'm working on a rust bindings, and it basically works now. Refer to the test which similiar to the C API

It's in very early stage, I'm refering to rusqlite to build this package. Anyone interested welcome to contribute.

https://github.com/wangfenjin/duckdb-rs

[wangfenjin]

Hi @hannesmuehleisen @Mytherin , currently for the C API, there is no step api for prepared statement, could you please help look into this?

it will be very useful for writing iterator style api

[hannes]

What do you mean by step API? Are you using the sqlite api wrapper? The C API does have support for prepared statements...

[wangfenjin]

@hannesmuehleisen Oh I'm using duckdb.h , see from here it has prepared statements.

Do you suggest me to use the sqlite api wrapper? I think about this before, but I think maybe in the future we will have many custom api so I use the original c api. But seems the c api is quite simple, and it's hard even impossible to use c++ api from rust

[wangfenjin]

It might be a good choice to use the sqlite api wrapper, as in that case we can reuse most code from rusqlite

[wangfenjin]

Do we have released version of binary library and header files for sqlite api wrapper? It will be helpful if we decided to use it in rust client

[Mytherin]

It might be a good choice to use the sqlite api wrapper, as in that case we can reuse most code from rusqlite

A potential problem with the SQLite API wrapper is that it is a C API that duplicates the sqlite3 API. This should work fine if we use dlopen/dlsym to read symbols (i.e. use a shared library), but could cause name clashes when both SQLite itself and the DuckDB + SQLite API wrappers are statically linked together. I am not sure how Rust handles linking, but if you go down that route I would first test if both DuckDB and SQLite can exist together in the same rust executable when the API wrappers are included.

Another problem with the SQLite API wrappers is that type information is lost in those wrappers, as SQLite is not a strongly typed system and has very few internal types. This might be undesirable, as the types that you could expose using those wrappers are very limited.

Do we have released version of binary library and header files for sqlite api wrapper? It will be helpful if we decided to use it in rust client

I believe we don't have this right now.

[ankoh]

I think the better solution would be the Arrow CDataInterface.
Arrow-rs can consume the CDataInterface without copies which would safe a lot of work for a duckdb-sys crate.

[wangfenjin]

ON CONFLICT DO NOTHING seems don't work when insert.

./duckdb
v0.2.7-dev775 7e1dd71df
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D CREATE TABLE bar(x INTEGER UNIQUE);
D INSERT INTO bar (x) VALUES (2) ON CONFLICT DO NOTHING;
D INSERT INTO bar (x) VALUES (2) ON CONFLICT DO NOTHING;
Error: Constraint Error: duplicate key value violates primary key or unique constraint

[hannes]

Indeed its likely that the ON CONFLICT NOTHING gets silently ignored at the moment. Could you open a separate issue for this please?

[wangfenjin]

Hi @Mytherin , thanks for your feedback.

1. I agree with you for the link issue. rust use the same link mechanism with c/c++, so rust will have same issue if user want to use sqlite and our wrapper at the same time.
2. For the type system problems, I think we should extend our current sqlite3 wrapper with more customized api, I'm thinking maybe we can have an extra header like duckdb3.h which includes sqlite3.h, and conditionally include and implement the api defined in duckdb3.h. That way we can not only expose the type system, and also other error code and utility functions like arrow or parquet support.
3. For the release issue, I suggest we can also conditionally bake the sqlite3 wrapper into the amalgamation code and binary, it will be much easier for us to depends on the features.

Overall I think we should have our own c api which adapt to our design, but at the same time the sqlite api had evolved for such a long time and widely used, it's a great advantage duckdb can have compatible api with it.

For the rust wrapper, for now I'll use the sqlite3 wrapper and we can migrate to the duckdb.h after it's get more feature. It's quite usable now I think, what I know is it seems doesn't have transaction and iterator style api like sqlite3_step.

[wangfenjin]

Hi all,

I had adapt rusqlite to work with duckdb, here are the minimum code changes. And we need to merge #1849 before these changes work.

If there are no other concerns, I will try to clean to the code and docs and release the package as duckdb-rs. After that we can based on this to add more duckdb specific features.

[Mytherin]

2. For the type system problems, I think we should extend our current sqlite3 wrapper with more customized api, I'm thinking maybe we can have an extra header like duckdb3.h which includes sqlite3.h, and conditionally include and implement the api defined in duckdb3.h. That way we can not only expose the type system, and also other error code and utility functions like arrow or parquet support.

I would prefer not to extend the sqlite api wrappers, but instead extend the C API itself. The main purpose of the sqlite API is to enable DuckDB to be used with programs that were written against the SQLite API itself. If additional functionality is required beyond the SQLite API it is better to use our own API rather than having a mix of the SQLite API and some additional API calls.

3. For the release issue, I suggest we can also conditionally bake the sqlite3 wrapper into the amalgamation code and binary, it will be much easier for us to depends on the features.

Yes, I am fine with that. We could add an #ifdef DUCKDB_SQLITE_API_WRAPPER in duckdb.cpp that enables building the SQLite API wrapper. This define can then be toggled by whoever builds the amalgamation.

[wangfenjin]

I would prefer not to extend the sqlite api wrappers, but instead extend the C API itself.

Agree, actually the sqlite api itself is extensible enough to let us inject / hook features we want, as long as we can support the extension related api. But I think we can add some constants to indicate the type as duckdb is static typing, as currently there are too few types in the sqlite api. It should only change the return value of this function with no other side effect.

[Mytherin]

Indeed a lot of type information is lost, but this is by design since the API tries to be compatible with SQLite itself (which does not have this type information). I think once you go down the road of customizing the API it is better to rewrite against the C/C++ API. The SQLite API is slow and not really intended for bulk data consumption/transfer. It would be cleaner and faster to use the C API and extend that as required.

[wangfenjin]

Agree.

The C API design is a thing that I don't think I have the ability to do. Do you have any plans about it? Maybe you guys can define the API / requirements, I can help on the implementation.

[sdmcallister]

@wangfenjin did you use bindgen to wrap the C api?

[wangfenjin]

@sdmcallister yes, you can refer to here https://github.com/wangfenjin/duckdb-rs/tree/main/libduckdb-sys

I haven't publish it, still working on it

[wangfenjin]

Initial version should be ready next week

[wangfenjin]

In case anyone interested, I had push my draft version to the master, still a lot of todos

[alanwilter]

I'm trying to compile my app, in Windows, with crate duckdb but it fails with:

= note: LINK : fatal error LNK1181: cannot open input file 'duckdb.lib'

Using Cargo.toml: duckdb = "0.10.2"

[Mause]

I'm trying to compile my app, in Windows, with crate duckdb but it fails with:

= note: LINK : fatal error LNK1181: cannot open input file 'duckdb.lib'

Using Cargo.toml: duckdb = "0.10.2"

If you don't want to provide your own copy of the DuckDB compiled library (duckdb.dll in your case), you can use the "bundled" feature.

In the future though, please direct rust related questions to https://github.com/duckdb/duckdb-rs