Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KeyError: 0 error with common-sql version 1.3.0 #27978

Closed
1 of 2 tasks
park-peter opened this issue Nov 29, 2022 · 6 comments · Fixed by #28006
Closed
1 of 2 tasks

KeyError: 0 error with common-sql version 1.3.0 #27978

park-peter opened this issue Nov 29, 2022 · 6 comments · Fixed by #28006
Labels
area:providers duplicate Issue that is duplicated kind:bug This is a clearly a bug priority:medium Bug that should be fixed before next release but would not block a release

Comments

@park-peter
Copy link

park-peter commented Nov 29, 2022 •
edited
Loading

Apache Airflow Provider(s)

common-sql

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==6.0.0
apache-airflow-providers-apache-hive==4.0.1
apache-airflow-providers-apache-livy==3.1.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.4.0
apache-airflow-providers-common-sql==1.3.0
apache-airflow-providers-databricks==3.3.0
apache-airflow-providers-dbt-cloud==2.2.0
apache-airflow-providers-elasticsearch==4.2.1
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.4.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-microsoft-azure==4.3.0
apache-airflow-providers-postgres==5.2.2
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-sftp==4.1.0
apache-airflow-providers-snowflake==3.3.0
apache-airflow-providers-sqlite==3.2.1
apache-airflow-providers-ssh==3.2.0

Apache Airflow version

2.4.3

Operating System

Debian Bullseye

Deployment

Astronomer

Deployment details

No response

What happened

With the latest version of common-sql provider, the get_records from hook is now a ordinary dictionary, causing this KeyError with SqlSensor:

[2022-11-29, 00:39:18 UTC] {taskinstance.py:1851} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/sensors/base.py", line 189, in execute
    poke_return = self.poke(context)
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/common/sql/sensors/sql.py", line 98, in poke
    first_cell = records[0][0]
KeyError: 0

I have only tested with Snowflake, I haven't tested it with other databases. Reverting back to 1.2.0 solves the issue.

What you think should happen instead

It should return an iterable list as usual with the query.

How to reproduce

from datetime import datetime

from airflow import DAG
from airflow.providers.common.sql.sensors.sql import SqlSensor

with DAG(
    dag_id="sql_provider_snowflake_test",
    schedule=None,
    start_date=datetime(2022, 1, 1),
    catchup=False,
):
    t1 = SqlSensor(
        task_id="snowflake_test",
        conn_id="snowflake",
        sql="select 0",
        fail_on_empty=False,
        poke_interval=20,
        mode="poke",
        timeout=60 * 5,
    )

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@park-peter park-peter added area:providers kind:bug This is a clearly a bug labels Nov 29, 2022
@potiuk
Copy link
Member

potiuk commented Nov 29, 2022 •
edited
Loading

Can you please try 1.3.1rc3 (it is going to be released today most likely)

@potiuk potiuk mentioned this issue Nov 29, 2022
Closed
2 tasks
@park-peter
Copy link
Author

Can you please try 1.3.1rc3 (it is going to be released today most likely)

The issue unfortunately continues to exist with 1.3.1.

@potiuk
Copy link
Member

potiuk commented Nov 29, 2022

Yeah I was afraid of it.

@potiuk potiuk added the priority:medium Bug that should be fixed before next release but would not block a release label Nov 29, 2022
@potiuk potiuk mentioned this issue Nov 29, 2022
Merged
@potiuk
Copy link
Member

potiuk commented Nov 30, 2022

This is basically duplicate of #27976. We should fix it in the same way. Closing this one as duplicate.

@potiuk potiuk closed this as completed Nov 30, 2022
@potiuk potiuk added the duplicate Issue that is duplicated label Nov 30, 2022
@potiuk potiuk mentioned this issue Nov 30, 2022
Merged
potiuk added a commit to potiuk/airflow that referenced this issue Nov 30, 2022
@potiuk
Make Snowflake Hook conform to semantics of DBApi
4ff1099 
This is a breaking change, that changes return value of the Hook,
however it keeps compatibility with regards of the value returned
by the SnowflakeOperator (so DAGs using the operator should be
unaffected).

You can also switch back to the Dictionary returned by Hooks by
specifying ``return_dictionaries`` parameter to run method of
the Hook.

At the same time SnowflakeHook can be used in a number of generic
operators and sensors such as:

* SQLColumnCheckOperator
* SQLSensor

Fixes: apache#27978
Fixes: apache#27976
potiuk added a commit that referenced this issue Dec 1, 2022
@potiuk
Make Snowflake Hook conform to semantics of DBApi (#28006)
d9cefcd 
* Make Snowflake Hook conform to semantics of DBApi

This is a breaking change, that changes return value of the Hook,
however it keeps compatibility with regards of the value returned
by the SnowflakeOperator (so DAGs using the operator should be
unaffected).

You can also switch back to the Dictionary returned by Hooks by
specifying ``return_dictionaries`` parameter to run method of
the Hook.

At the same time SnowflakeHook can be used in a number of generic
operators and sensors such as:

* SQLColumnCheckOperator
* SQLSensor

Fixes: #27978
Fixes: #27976
@potiuk
Copy link
Member

potiuk commented Dec 1, 2022 •
edited
Loading

@park-peter - can you please test the new snowflake 4.0.2rc1 and report it in the issue here: #27978
(all details are there)

jrggggg pushed a commit to jrggggg/airflow that referenced this issue Dec 1, 2022
@potiuk @jrggggg
Make Snowflake Hook conform to semantics of DBApi (apache#28006)
a777fac 
* Make Snowflake Hook conform to semantics of DBApi

This is a breaking change, that changes return value of the Hook,
however it keeps compatibility with regards of the value returned
by the SnowflakeOperator (so DAGs using the operator should be
unaffected).

You can also switch back to the Dictionary returned by Hooks by
specifying ``return_dictionaries`` parameter to run method of
the Hook.

At the same time SnowflakeHook can be used in a number of generic
operators and sensors such as:

* SQLColumnCheckOperator
* SQLSensor

Fixes: apache#27978
Fixes: apache#27976
@park-peter
Copy link
Author

@potiuk with snowflake 4.0.2rc1 and common-sql 1.3.1, I can confirm that things look good!

@potiuk potiuk mentioned this issue Dec 1, 2022
Closed
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:providers duplicate Issue that is duplicated kind:bug This is a clearly a bug priority:medium Bug that should be fixed before next release but would not block a release
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make Snowflake Hook conform to semantics of DBApi potiuk/airflow
2 participants
@potiuk @park-peter