Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

duckdb as DataSource #14563

Closed
ravikishorer opened this issue May 11, 2021 · 9 comments
Closed

duckdb as DataSource #14563

ravikishorer opened this issue May 11, 2021 · 9 comments
Assignees
@srinify

Comments

@ravikishorer
Copy link

Is your feature request related to a problem? Please describe.
To be able to connect to Duckdb as datasource.

Describe the solution you'd like
To be able to connect to Duckdb just like how easy it is to connect to sqlite currently.

Describe alternatives you've considered
Sqlite is an alternative. But as Duckdb is more performant for analytical use cases, it will be a great addition.

Additional context
DuckDB is an in-process SQL OLAP database management system - https://duckdb.org/. It is much faster than sqlite for analytical purposes.

From https://duckdb.org/docs/api/python,
The standard DuckDB Python API provides a SQL interface compliant with the DB-API 2.0 specification described by PEP 249 similar to the SQLite Python API.
@srinify srinify self-assigned this May 11, 2021
@srinify
Copy link
Contributor

srinify commented May 11, 2021 •
edited

hi @ravikishorer this is cool! Do you know how mature the sqlalchemy dialect is for duckdb? This thread seems to imply its still very early: duckdb/duckdb#305

I'm happy to help here

@ravikishorer
Copy link
Author

@srinify I'm not fully aware. Based on the documentation, it is based on postgres dialect and in the issue you have attached, someone has implemented a very basic engine wrapping postgres one - https://pypi.org/project/duckdb-engine/

@Mause
Copy link

Mause commented May 27, 2021

That someone was me - if you have any questions or suggestions for how I have implemented it, feel free to tag me :)

@mpope9 mpope9 mentioned this issue Aug 1, 2021
Open
@alitrack
Copy link
Contributor

I forked both duckdb-engine and superset, and made superset support duckdb now.

duckdb0

db_engine_specs/duckdb.py is simply copy and modify from db_engine_specs/sqlite.py

and here is the video https://www.bilibili.com/video/BV1zQ4y1q7ti/

@terekete
Copy link

terekete commented Jan 8, 2022

Q. With DuckDB I see its has some support for Apache Arrow. Is there some capabilities between DuckDB and Superset to take advantage of arrow format ?

@armando-fandango
Copy link
Contributor

I forked both duckdb-engine and superset, and made superset support duckdb now.

db_engine_specs/duckdb.py is simply copy and modify from db_engine_specs/sqlite.py

and here is the video https://www.bilibili.com/video/BV1zQ4y1q7ti/

@alitrack Did you send the pull request in for your solution for duckdb please ?

@alitrack
Copy link
Contributor

alitrack commented Feb 2, 2022

I forked both duckdb-engine and superset, and made superset support duckdb now.
db_engine_specs/duckdb.py is simply copy and modify from db_engine_specs/sqlite.py
and here is the video https://www.bilibili.com/video/BV1zQ4y1q7ti/

@alitrack Did you send the pull request in for your solution for duckdb please ?

yes, I did.

@alitrack
Copy link
Contributor

alitrack commented Mar 5, 2022

@armando-fandango my PR is merged.

@rwhaling rwhaling mentioned this issue Mar 20, 2022
Closed
9 tasks
@rwhaling
Copy link
Contributor

took the liberty of opening up a PR from @alitrack's fork, since this is awesome and it looks like the sqlalchemy side is unblocked.

@rwhaling rwhaling mentioned this issue Mar 22, 2022
Merged
9 tasks
villebro pushed a commit that referenced this issue Apr 3, 2022
@rwhaling @villebro
feat: add duckdb as DataSource - Fixes #14563 (#19317)
89b7b37 
* + duckdb support

needs the forked version of [duckdb-engine](https://github.com/alitrack/duckdb_engine)

* Update duckdb.py

update  _time_grain_expressions

* removed superfluous get_all_datasource_names def in duckdb engine spec

* added exception handling for duckdb single-threaded RuntimeError

* fixed linter blips and other stylistic cleanup in duckdb.py

* one last round of linter tweaks in test_connection.py for duckdb support

Co-authored-by: Steven Lee <admin@alitrack.com>
Co-authored-by: Richard Whaling <richardwhaling@Richards-MacBook-Pro.local>
(cherry picked from commit 202e34a)
philipher29 pushed a commit to ValtechMobility/superset that referenced this issue Jun 9, 2022
@rwhaling @alitrack @philipher29
feat: add duckdb as DataSource - Fixes apache#14563 (apache#19317)
01830ba 
* + duckdb support

needs the forked version of [duckdb-engine](https://github.com/alitrack/duckdb_engine)

* Update duckdb.py

update  _time_grain_expressions

* removed superfluous get_all_datasource_names def in duckdb engine spec

* added exception handling for duckdb single-threaded RuntimeError

* fixed linter blips and other stylistic cleanup in duckdb.py

* one last round of linter tweaks in test_connection.py for duckdb support

Co-authored-by: Steven Lee <admin@alitrack.com>
Co-authored-by: Richard Whaling <richardwhaling@Richards-MacBook-Pro.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@srinify srinify

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@rwhaling @armando-fandango @srinify @Mause @terekete @alitrack @ravikishorer