Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove rust dependency from Cosmos #530

Closed
tatiana opened this issue Sep 12, 2023 · 2 comments · Fixed by #526
Closed

Remove rust dependency from Cosmos #530

tatiana opened this issue Sep 12, 2023 · 2 comments · Fixed by #526
Labels
team/astro-devex
Milestone
1.1.1

Comments

@tatiana
Copy link
Collaborator

tatiana commented Sep 12, 2023

To install Cosmos, users must install Rust as part of the recently introduced openlineage-integration-common package dependency, even if they don't opt to use openlineage.

The user ES reported this bug in the Slack channel:
https://apache-airflow.slack.com/archives/C059CC42E9W/p1694525966501819

"Hi

pip install astronomer-cosmos==1.1.0

gives me:

Collecting openlineage-sql==1.1.0 (from openlineage-integration-common->astronomer-cosmos==1.1.0->-r r.txt (line 2))
  Using cached openlineage_sql-1.1.0.tar.gz (31 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]

      Cargo, the Rust package manager, is not installed or is not on PATH.
      This package requires Rust and Cargo to compile extensions. Install it through
      the system's package manager or via https://rustup.rs/

      Checking for Rust toolchain....
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
@tatiana tatiana added this to the 1.1.1 milestone Sep 12, 2023
@tatiana tatiana mentioned this issue Sep 13, 2023
Merged
tatiana added a commit that referenced this issue Sep 14, 2023
@tatiana
Fix issue reported by the community in slack with lineage in Cosmos 1…
665a9b8 
….1 (#526)

Fix the following issue, that was reported in
[slack](https://apache-airflow.slack.com/archives/C059CC42E9W/p1694212778764869)
by a user named Sai:

"After upgrading to cosmos 1.1, I see the below error in Airflow logs. I
don't have any OpenLineage integrations. Is there a way to stop emitting
this error?"

```
[2023-09-08, 22:28:07 UTC] {taskinstance.py:1318} INFO - Marking task as SUCCESS. dag_id=high_freq_core_dbt, task_id=dbt_front.daily_company_user_metrics.daily_company_user_metrics_run, execution_date=20230908T183000, start_date=20230908T222730, end_date=20230908T222807
[2023-09-08, 22:28:07 UTC] {manager.py:61} ERROR - Failed to extract metadata 'TaskInstance' object has no attribute 'openlineage_events_completes' task_type=DbtRunLocalOperator airflow_dag_id=high_freq_core_dbt task_id=dbt_front.daily_company_user_metrics.daily_company_user_metrics_run airflow_run_id=scheduled__2023-09-08T18:30:00+00:00 
Traceback (most recent call last):
  File "/home/airflow/python-venvs/data-pipelines/lib/python3.10/site-packages/openlineage/airflow/extractors/manager.py", line 45, in extract_metadata
    task_metadata = extractor.extract_on_complete(task_instance)
  File "/home/airflow/python-venvs/data-pipelines/lib/python3.10/site-packages/openlineage/airflow/extractors/base.py", line 116, in extract_on_complete
    return self._get_openlineage_facets(
  File "/home/airflow/python-venvs/data-pipelines/lib/python3.10/site-packages/openlineage/airflow/extractors/base.py", line 124, in _get_openlineage_facets
    facets: OperatorLineage = get_facets_method(*args)
  File "/home/airflow/python-venvs/data-pipelines/lib/python3.10/site-packages/cosmos/operators/local.py", line 318, in get_openlineage_facets_on_complete
    for completed in task_instance.openlineage_events_completes:
AttributeError: 'TaskInstance' object has no attribute 'openlineage_events_completes'
```

The issue was reported to happen using:
* Airflow version: v2.5.1
* dbt version: 1.5.1 
* without any intentional setup of OpenLineage

Since the issue was reported, I learned that
`get_openlineage_facets_on_complete` is called even if `execute` fails
because `get_openlineage_facets_on_failure` is not currently
implemented.

Closes: #530 (Rust dependency issue reported)
@tatiana
Copy link
Collaborator Author

tatiana commented Sep 14, 2023

We identified that the user was running Cosmos on Windows, and in these cases, the openlineage-sql library, which is a dependency of openlineage-integration-comon, didn't have a pre-compiled Rust for it.

@tatiana tatiana mentioned this issue Sep 14, 2023
Open
@tatiana
Copy link
Collaborator Author

tatiana commented Sep 14, 2023

For now, we removed openlineage-integration-comon as a requirement of Cosmos. We can review once/if OpenLineage/OpenLineage#2110 is implemented.

As discussed in Slack, there is no Airflow Window support) yet, but things may change.

We know many dbt users develop on Windows, and from a Cosmos perspective, we can support this user-journey, where possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
team/astro-devex
Projects
None yet
Milestone
1.1.1
Development

Successfully merging a pull request may close this issue.

Fix issue reported by the community in slack with lineage in Cosmos 1.1 astronomer/astronomer-cosmos
1 participant
@tatiana