FR: implement Arrow Flight SQL

[domoritz]

https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/.

Flight SQL is a new client-server protocol developed by the Apache Arrow community for interacting with SQL databases that makes use of the Arrow in-memory columnar format and the Flight RPC framework.

It would make a lot of sense for DuckDB since flight SQL works best with columnar databases like DuckDB.

Here is a client implementation for SQLite: https://github.com/apache/arrow/blob/release-7.0.0/cpp/src/arrow/flight/sql/example/sqlite_server.cc

[hamilton]

This would be fantastic, especially if it were at the engine / extension level & not solely for Python & R. I hope that in general the programmatic case w/ multiple clients can be supported. I know the arrow support in duckdb seems to be restricted to Python & R today and this is already kind of a bummer for data application development.

[domoritz]

https://github.com/duckdb/duckdb-wasm also uses Arrow to move data between the wasm and the js contexts.

[wangfenjin]

Arrow is also supported in c-api, so any language that can interact with C API should be able to support arrow

[wangfenjin]

As duckdb tries to avoid external dependencies, not sure if it's easy to support flight sql in duckdb core. Maybe it's better to be support as an extension or a 3rd party component

[hannes]

I think it makes more sense for us to implement ADBC: https://docs.google.com/document/d/1t7NrC76SyxL_OffATmjzZs2xcj1owdUsIF2WKL_Zw1U/edit?usp=sharing

There is a standalone header we would be able to vendor:
https://github.com/lidavidm/arrow/blob/adbc/cpp/src/adbc/adbc.h

[prmoore77]

Hi - we have implemented an example Flight SQL Server with DuckDB as a back-end here: https://github.com/voltrondata/flight-sql-server-example

Contributions are welcome!