-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Transform op - database is not picked from metadata #1034
Comments
This looks like a problem when using snowflake. I quickly tested the bigquery worked fine |
For the hook creation in astro-sdk we tightly depends on the airflow connection
Snowflake hook expects the database, schema, etc., so the ideal place to set these values is a connection only. Even in our case, we do
If a user is setting up the schema, and database in the input and output table in the transform operator which value should we use from input_table or output_table or will we create multiple hooks? it looks a little bit complicated to me in the current design. For now, I have modified the |
# Description closes: #1034 Add table param in ```create_database``` function so that can access and use table metadata while creating a hook ## What is the current behavior? Currently, we don't use Table metadata param schema and database while creating so if these values is not set in the airflow connection we are getting an error in the case of snowflake ## What is the new behavior? change ``` def create_database(conn_id: str) -> BaseDatabase: ... ``` to ``` def create_database( conn_id: str, table: object | None = None, ) -> BaseDatabase: ... ``` and use table metadata if available while creating snowflake hook ## Does this introduce a breaking change? No ### Checklist - [ ] Created tests which fail without the change (if possible) - [ ] Extended the README / documentation, if necessary Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
# Description closes: #1034 Add table param in ```create_database``` function so that can access and use table metadata while creating a hook ## What is the current behavior? Currently, we don't use Table metadata param schema and database while creating so if these values is not set in the airflow connection we are getting an error in the case of snowflake ## What is the new behavior? change ``` def create_database(conn_id: str) -> BaseDatabase: ... ``` to ``` def create_database( conn_id: str, table: object | None = None, ) -> BaseDatabase: ... ``` and use table metadata if available while creating snowflake hook ## Does this introduce a breaking change? No ### Checklist - [ ] Created tests which fail without the change (if possible) - [ ] Extended the README / documentation, if necessary Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Describe the bug
snowflake.connector.errors.ProgrammingError: 090105 (22000): 01a77af0-0605-96b0-0000-682112f6e8ae: Cannot perform SELECT. This session does not have a current database. Call 'USE DATABASE', or use a qualified name.
ref: https://astronomer.slack.com/archives/C02B8SPT93K/p1665414029917169
Version
To Reproduce
Try the above code.
Expected behavior
If the DB is not specified in the connection we should be able to pick from metadata.
The text was updated successfully, but these errors were encountered: