You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
On a successful task completion with the @aql.transform() decorator, extraction is failing when utilizing the astro.lineage.extractor.PythonSDKExtractor.
Version
Astro: 1.2.0
astro-runtime:6.0.2
To Reproduce
Steps to reproduce the behavior:
Create a connection to a Snowflake Instance.
Write a DAG using the @aql.transform() operator, utilizing this connection.
Expected behavior
The extractor is able to find the conn_id of the task.
Logs
[2022-10-26, 02:40:59 UTC] {taskinstance.py:1401} INFO - Marking task as SUCCESS. dag_id=DAG_ID, task_id=TASK_ID, execution_date=20221021T202905, start_date=20221026T024051, end_date=20221026T024059
[2022-10-26, 02:40:59 UTC] {manager.py:63} ERROR - Failed to extract metadata The conn_id `` isn't defined task_type=TransformOperator airflow_dag_id=DAG_ID task_id=TASK_ID airflow_run_id=manual__2022-10-21T20:29:05.395735+00:00
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/openlineage/airflow/extractors/manager.py", line 46, in extract_metadata
task_metadata = extractor.extract_on_complete(task_instance)
File "/usr/local/lib/python3.9/site-packages/astro/lineage/extractor.py", line 59, in extract_on_complete
open_lineage_facets: OpenLineageFacets = self.operator.get_openlineage_facets(task_instance)
File "/usr/local/lib/python3.9/site-packages/astro/sql/operators/base_decorator.py", line 203, in get_openlineage_facets
f"{self.output_table.openlineage_dataset_namespace()}"
File "/usr/local/lib/python3.9/site-packages/astro/table.py", line 173, in openlineage_dataset_namespace
database = create_database(self.conn_id)
File "/usr/local/lib/python3.9/site-packages/astro/databases/__init__.py", line 30, in create_database
conn_type = BaseHook.get_connection(conn_id).conn_type
File "/usr/local/lib/python3.9/site-packages/airflow/hooks/base.py", line 70, in get_connection
conn = Connection.get_connection_from_secrets(conn_id)
File "/usr/local/lib/python3.9/site-packages/airflow/models/connection.py", line 432, in get_connection_from_secrets
raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.exceptions.AirflowNotFoundException: The conn_id `` isn't defined
[2022-10-26, 02:40:59 UTC] {local_task_job.py:164} INFO - Task exited with return code 0
[2022-10-26, 02:40:59 UTC] {local_task_job.py:273} INFO - 0 downstream tasks scheduled from follow-on schedule check
The text was updated successfully, but these errors were encountered:
# Description
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
`conn_id` is missing in output table for transform operator. It should
be checked for that
<!--
Issues are required for both bug fixes and features.
Reference it using one of the following:
closes: #ISSUE
related: #ISSUE
-->
closes: #1152
## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->
- Check if `conn_id` is empty
- Push row count of output table to xcom
- Fixed some flaky test
## Does this introduce a breaking change?
No
### Checklist
- [x] Created tests which fail without the change (if possible)
- [x] Extended the README / documentation, if necessary
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: rajaths010494 <rajath.srinivasaiah@astronomer.io>
Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>
Co-authored-by: Pankaj <pankaj.singh@astronomer.io>
# Description
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
`conn_id` is missing in output table for transform operator. It should
be checked for that
<!--
Issues are required for both bug fixes and features.
Reference it using one of the following:
closes: #ISSUE
related: #ISSUE
-->
closes: #1152
## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->
- Check if `conn_id` is empty
- Push row count of output table to xcom
- Fixed some flaky test
## Does this introduce a breaking change?
No
### Checklist
- [x] Created tests which fail without the change (if possible)
- [x] Extended the README / documentation, if necessary
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: rajaths010494 <rajath.srinivasaiah@astronomer.io>
Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>
Co-authored-by: Pankaj <pankaj.singh@astronomer.io>
Describe the bug
On a successful task completion with the
@aql.transform()
decorator, extraction is failing when utilizing theastro.lineage.extractor.PythonSDKExtractor
.Version
To Reproduce
Steps to reproduce the behavior:
@aql.transform()
operator, utilizing this connection.Expected behavior
The extractor is able to find the
conn_id
of the task.Logs
The text was updated successfully, but these errors were encountered: