Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conn ID not found on Extraction with the @aql.transform() Decorator #1152

Closed
ReadytoRocc opened this issue Oct 28, 2022 · 0 comments · Fixed by #1181
Closed

Conn ID not found on Extraction with the @aql.transform() Decorator #1152

ReadytoRocc opened this issue Oct 28, 2022 · 0 comments · Fixed by #1181
Assignees
Labels
product/python-sdk Label describing products
Milestone

Comments

@ReadytoRocc
Copy link

Describe the bug
On a successful task completion with the @aql.transform() decorator, extraction is failing when utilizing the astro.lineage.extractor.PythonSDKExtractor.

Version

  • Astro: 1.2.0
  • astro-runtime:6.0.2

To Reproduce
Steps to reproduce the behavior:

  1. Create a connection to a Snowflake Instance.
  2. Write a DAG using the @aql.transform() operator, utilizing this connection.

Expected behavior
The extractor is able to find the conn_id of the task.

Logs

[2022-10-26, 02:40:59 UTC] {taskinstance.py:1401} INFO - Marking task as SUCCESS. dag_id=DAG_ID, task_id=TASK_ID, execution_date=20221021T202905, start_date=20221026T024051, end_date=20221026T024059
[2022-10-26, 02:40:59 UTC] {manager.py:63} ERROR - Failed to extract metadata The conn_id `` isn't defined task_type=TransformOperator airflow_dag_id=DAG_ID task_id=TASK_ID airflow_run_id=manual__2022-10-21T20:29:05.395735+00:00 
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/openlineage/airflow/extractors/manager.py", line 46, in extract_metadata
    task_metadata = extractor.extract_on_complete(task_instance)
  File "/usr/local/lib/python3.9/site-packages/astro/lineage/extractor.py", line 59, in extract_on_complete
    open_lineage_facets: OpenLineageFacets = self.operator.get_openlineage_facets(task_instance)
  File "/usr/local/lib/python3.9/site-packages/astro/sql/operators/base_decorator.py", line 203, in get_openlineage_facets
    f"{self.output_table.openlineage_dataset_namespace()}"
  File "/usr/local/lib/python3.9/site-packages/astro/table.py", line 173, in openlineage_dataset_namespace
    database = create_database(self.conn_id)
  File "/usr/local/lib/python3.9/site-packages/astro/databases/__init__.py", line 30, in create_database
    conn_type = BaseHook.get_connection(conn_id).conn_type
  File "/usr/local/lib/python3.9/site-packages/airflow/hooks/base.py", line 70, in get_connection
    conn = Connection.get_connection_from_secrets(conn_id)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/connection.py", line 432, in get_connection_from_secrets
    raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.exceptions.AirflowNotFoundException: The conn_id `` isn't defined
[2022-10-26, 02:40:59 UTC] {local_task_job.py:164} INFO - Task exited with return code 0
[2022-10-26, 02:40:59 UTC] {local_task_job.py:273} INFO - 0 downstream tasks scheduled from follow-on schedule check
@sunank200 sunank200 self-assigned this Nov 2, 2022
@pankajastro pankajastro added the product/python-sdk Label describing products label Nov 2, 2022
@phanikumv phanikumv added this to the 1.2.1 milestone Nov 4, 2022
pankajastro added a commit that referenced this issue Nov 4, 2022
# Description
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
`conn_id` is missing in output table for transform operator. It should
be checked for that

<!--
Issues are required for both bug fixes and features.
Reference it using one of the following:

closes: #ISSUE
related: #ISSUE
-->
closes: #1152


## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->

- Check if `conn_id` is empty
- Push row count of output table to xcom
- Fixed some flaky test

## Does this introduce a breaking change?
No

### Checklist
- [x] Created tests which fail without the change (if possible)
- [x] Extended the README / documentation, if necessary

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: rajaths010494 <rajath.srinivasaiah@astronomer.io>
Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>
Co-authored-by: Pankaj <pankaj.singh@astronomer.io>
utkarsharma2 pushed a commit that referenced this issue Nov 4, 2022
# Description
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
`conn_id` is missing in output table for transform operator. It should
be checked for that

<!--
Issues are required for both bug fixes and features.
Reference it using one of the following:

closes: #ISSUE
related: #ISSUE
-->
closes: #1152


## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->

- Check if `conn_id` is empty
- Push row count of output table to xcom
- Fixed some flaky test

## Does this introduce a breaking change?
No

### Checklist
- [x] Created tests which fail without the change (if possible)
- [x] Extended the README / documentation, if necessary

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: rajaths010494 <rajath.srinivasaiah@astronomer.io>
Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>
Co-authored-by: Pankaj <pankaj.singh@astronomer.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
product/python-sdk Label describing products
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants