-
Notifications
You must be signed in to change notification settings - Fork 127
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix Dataset URI by using openlineage-integration-common as lib #485
Conversation
👷 Deploy Preview for amazing-pothos-a3bca0 processing.
|
ad38234
to
0845e12
Compare
0845e12
to
c0d5698
Compare
c0d5698
to
472159a
Compare
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #485 +/- ##
==========================================
- Coverage 91.51% 91.48% -0.04%
==========================================
Files 50 49 -1
Lines 1874 1914 +40
==========================================
+ Hits 1715 1751 +36
- Misses 159 163 +4
☔ View full report in Codecov by Sentry. |
2008dc9
to
e02be51
Compare
Thanks a lot for the review and feedback, @harels @mobuchowski and @JDarDagran ! I've addressed most of the feedback. As @harels mentioned, one point that is missing is to have a better automated testing strategy to make sure Cosmos is emitting the open lineage events correctly. I've logged a follow up ticket for us to improve this: #509 509 |
As observed in: #482 The following exception was raised: build of https://github.com/pre-commit/mirrors-mypy:types-PyYAML,types-attrs,attrs,types-requests,types-python-dateutil,apache-airflow@v1.5.1 for python@python3 exceeds tier max size 250MiB: 262.2MiB So we disabled MyPy checks as part of #485
As observed in: #482 The following exception was raised: build of https://github.com/pre-commit/mirrors-mypy:types-PyYAML,types-attrs,attrs,types-requests,types-python-dateutil,apache-airflow@v1.5.1 for python@python3 exceeds tier max size 250MiB: 262.2MiB So we disabled MyPy checks as part of #485
As observed in: #482 The following exception was raised: build of https://github.com/pre-commit/mirrors-mypy:types-PyYAML,types-attrs,attrs,types-requests,types-python-dateutil,apache-airflow@v1.5.1 for python@python3 exceeds tier max size 250MiB: 262.2MiB So we disabled MyPy checks as part of #485
**Features** * Support dbt global flags (via dbt_cmd_global_flags in operator_args) by @tatiana in #469 * Support parsing DAGs when there are no connections by @jlaneve in #489 **Enhancements** * Hide sensitive field when using BigQuery keyfile_dict profile mapping by @jbandoro in #471 * Consistent Airflow Dataset URIs, inlets and outlets with `Openlineage package <https://pypi.org/project/openlineage-integration-common/>`_ by @tatiana in #485. `Read more <https://astronomer.github.io/astronomer-cosmos/configuration/lineage.html>`_. * Refactor ``LoadMethod.DBT_LS`` to run from a temporary directory with symbolic links by @tatiana in #488 * Run ``dbt deps`` when using ``LoadMethod.DBT_LS`` by @DanMawdsleyBA in #481 * Update Cosmos log color to purple by @harels in #494 * Change operators to log ``dbt`` commands output as opposed to recording to XCom by @tatiana in #513 **Bug fixes** * Fix bug on select node add exclude selector subset ids logic by @jensenity in #463 * Refactor dbt ls to run from a temporary directory, to avoid Read-only file system errors during DAG parsing, by @tatiana in #414 * Fix profile_config arg in DbtKubernetesBaseOperator by @david-mag in #505 * Fix SnowflakePrivateKeyPemProfileMapping private_key reference by @nacpacheco in #501 * Fix incorrect temporary directory creation in VirtualenvOperator init by @tatiana in #500 * Fix log propagation issue by @tatiana in #498 * Fix PostgresUserPasswordProfileMapping to retrieve port from connection by @jlneve in #511 **Others** * Docs: Fix RenderConfig load argument by @jbandoro in #466 * Enable CI integration tests from external forks by @tatiana in #458 * Improve CI tests runtime by @tatiana in #457 * Change CI to run coverage after tests pass by @tatiana in #461 * Fix forks code revision in code coverage by @tatiana in #472 * [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #467 * Drop support to Python 3.7 in the CI test matrix by @harels in #490 * Add Airflow 2.7 to the CI test matrix by @tatiana in #487 * Add MyPy type checks to CI since we exceeded pre-commit disk quota usage by @tatiana in #510
Use
openlineage-dbt
to create outlets Dataset URIs from within Cosmos.Closes: #305
Closes: #497
Closes: #433 (only emits outlet events from the model - the same behaviour as openlineage-dbt)
Validation
This change was tested by running Marquez locally and triggering the dag
basic_cosmos_dag
using Airflow 2.6.1 and Python 3.10.10.The output generated by this version of Cosmos can be seen in the following screenshots:
Tasks
Emit open lineage eventsSupport Docker/K8s(deverred to issue: Add support for openlineage when running from Kubernetes/Docker #496)Create a PR on openlineage-dbt to remove the dbt dependencyNot needed since the code depends onopenlineage-integration-common
and notopenlineage-dbt
(more info)