Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPIKE/RESEARCH]:Add Duck DB support #1138

Closed
phanikumv opened this issue Oct 26, 2022 · 6 comments · Fixed by #1695
Closed

[SPIKE/RESEARCH]:Add Duck DB support #1138

phanikumv opened this issue Oct 26, 2022 · 6 comments · Fixed by #1695
Assignees
@sunank200
Labels
product/python-sdk Label describing products
Milestone
1.5.0

Comments

@phanikumv
Copy link
Collaborator

No description provided.

@phanikumv phanikumv added this to the 1.4.0 milestone Oct 26, 2022
@pankajastro pankajastro added the product/python-sdk Label describing products label Oct 26, 2022
@kaxil
Copy link
Collaborator

kaxil commented Nov 7, 2022

@jlaneve has built a provider for DuckDB on https://github.com/astronomer/airflow-provider-duckdb which is released on https://pypi.org/project/airflow-provider-duckdb/

@pankajastro
Copy link
Contributor

@jlaneve has built a provider for DuckDB on https://github.com/astronomer/airflow-provider-duckdb which is released on https://pypi.org/project/airflow-provider-duckdb/

Thank you @kaxil for sharing it. I have also started exploring it, I have a branch in our airflow repo for now but a bit of testing https://github.com/astronomer/airflow/tree/duckdb_provider

@utkarsharma2 utkarsharma2 modified the milestones: 1.4.0, 1.3.0 Nov 9, 2022
@sunank200 sunank200 modified the milestones: 1.3.0, 1.4.0 Nov 21, 2022
@phanikumv phanikumv assigned sunank200 and unassigned pankajastro Nov 30, 2022
@pankajastro pankajastro modified the milestones: 1.4.0, 1.5.0 Dec 22, 2022
@sunank200 sunank200 modified the milestone: 1.5.0 Jan 24, 2023
@sunank200
Copy link
Contributor

Duckdb doesn't seem to be part of external dialects of Sqlachemy: https://docs.sqlalchemy.org/en/20/dialects/ Also there is one library( https://pypi.org/project/duckdb-engine/ ) which acts as SQLAlchemy driver for DuckDB but this is not maintained well. In this case, all the methods for the base databases have to be overridden as is no SQLAlchemy support. This would increase the time for implementation and the tests.

cc: @utkarsharma2 @phanikumv

@sunank200
Copy link
Contributor

There is an issue already on duckdb: duckdb/duckdb#305

@sunank200
Copy link
Contributor

This is a PR on duckdb-engine for SQLAlchemy support for 2.0 But its very old

@sunank200 sunank200 mentioned this issue Feb 1, 2023
Merged
2 tasks
@sunank200
Copy link
Contributor

Used duckdb-engine for SQLALchemy driver as per discussion in https://astronomer.slack.com/archives/C03868KGF2Q/p1675274903489399

utkarsharma2 pushed a commit that referenced this issue Feb 8, 2023
@sunank200 @tatiana
@jlaneve @utkarsharma2
Add duckdb support (#1695)
da31b44 
# Description
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
Currently Duckdb is not supported with astro-sdk-python
<!--
Issues are required for both bug fixes and features.
Reference it using one of the following:

closes: #ISSUE
related: #ISSUE
-->
closes: #1130
closes: #1138

## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->
- Add duckdb support 
- Add the tests
- Update the documentation
- Add example DAG

## Does this introduce a breaking change?
No

### Checklist
- [x] Created tests which fail without the change (if possible)
- [x] Extended the README / documentation, if necessary

---------

Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Co-authored-by: Julian LaNeve <lanevejulian@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sunank200 sunank200

Labels
product/python-sdk Label describing products
Projects
None yet
Milestone
1.5.0
Development

Successfully merging a pull request may close this issue.

Add duckdb support astronomer/astro-sdk
5 participants
@sunank200 @kaxil @utkarsharma2 @phanikumv @pankajastro