python: use v2 Python facets #2693
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
extractor
area:integration/airflow
kacpermuda
added
area:integration/dbt
kacpermuda
approved these changes
May 14, 2024
integration/common/openlineage/common/provider/dbt/processor.py
Outdated
Show resolved
Hide resolved
JDarDagran
force-pushed
the
use-v2-facets
branch
4 times, most recently
from
9d62354
to
0ba663a
Compare
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #2693 +/- ##
==========================================
- Coverage 84.59% 83.81% -0.78%
==========================================
Files 59 54 -5
Lines 3446 3281 -165
==========================================
- Hits 2915 2750 -165
Misses 531 531 ☔ View full report in Codecov by Sentry. |
|
@kacpermuda @mobuchowski I've added |
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com> Use v2 facets in common. Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
Adjust bigquery URI to be valid URI. Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
boring-cyborg
bot
added
area:tests
mobuchowski
approved these changes
May 22, 2024
tatiana
added a commit
to astronomer/astronomer-cosmos
that referenced
this pull request
May 29, 2024
Fix three problems observed while running the tests in the CI recently: 1. Static check Fix spelling issue captured by static check (#1000) 2. Integration test that's no longer needed The issue we were trying to capture for no longer happens in the latest version of the Apache Airflow provider `apache-airflow-providers-postgres==5.11.1rc1`: apache/airflow#39842 3. Skip a buggy version of OL There was a breaking change between `openlineage-integration-common==1.14.0` and `openlineage-integration-common==1.15.0` . It may have been an unintended side-effect of OpenLineage/OpenLineage#2693. This is an example of how Cosmos was using `DbtLocalArtifactProcessor` - something that had been agreed upon in the past: ``` openlineage_processor = DbtLocalArtifactProcessor( producer=OPENLINEAGE_PRODUCER, job_namespace=LINEAGE_NAMESPACE, project_dir=project_dir, profile_name=self.profile_config.profile_name, target=self.profile_config.target_name, ) events = openlineage_processor.parse() for completed in events.completes: for output in getattr(completed, source): dataset_uri = output.namespace + "/" + output.name uris.append(dataset_uri) ``` In `openlineage-integration-common==1.14.0` and earlier versions, this would create URIs in the format: ``` postgres://0.0.0.0:5432/postgres.public.stg_customers ``` Since openlineage-integration-common==1.15.0 , this leads to URIs being created in the format: ``` postgres.public.stg_customers/postgres://0.0.0.0:5432 ``` This was fixed in OpenLineage/OpenLineage#2735 and released as part of OL 1.16: https://github.com/OpenLineage/OpenLineage/releases/tag/1.16.0
tatiana
added a commit
to astronomer/astronomer-cosmos
that referenced
this pull request
Jun 6, 2024
Fix three problems observed while running the tests in the CI recently: 1. Static check Fix spelling issue captured by static check (#1000) 2. Integration test that's no longer needed The issue we were trying to capture for no longer happens in the latest version of the Apache Airflow provider `apache-airflow-providers-postgres==5.11.1rc1`: apache/airflow#39842 3. Skip a buggy version of OL There was a breaking change between `openlineage-integration-common==1.14.0` and `openlineage-integration-common==1.15.0` . It may have been an unintended side-effect of OpenLineage/OpenLineage#2693. This is an example of how Cosmos was using `DbtLocalArtifactProcessor` - something that had been agreed upon in the past: ``` openlineage_processor = DbtLocalArtifactProcessor( producer=OPENLINEAGE_PRODUCER, job_namespace=LINEAGE_NAMESPACE, project_dir=project_dir, profile_name=self.profile_config.profile_name, target=self.profile_config.target_name, ) events = openlineage_processor.parse() for completed in events.completes: for output in getattr(completed, source): dataset_uri = output.namespace + "/" + output.name uris.append(dataset_uri) ``` In `openlineage-integration-common==1.14.0` and earlier versions, this would create URIs in the format: ``` postgres://0.0.0.0:5432/postgres.public.stg_customers ``` Since openlineage-integration-common==1.15.0 , this leads to URIs being created in the format: ``` postgres.public.stg_customers/postgres://0.0.0.0:5432 ``` This was fixed in OpenLineage/OpenLineage#2735 and released as part of OL 1.16: https://github.com/OpenLineage/OpenLineage/releases/tag/1.16.0
fafnirZ
pushed a commit
to fafnirZ/OpenLineage
that referenced
this pull request
Jul 3, 2024
* Use v2 facets in Airflow. Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com> Use v2 facets in common. Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com> * Python URI validator should expect scheme only. Adjust bigquery URI to be valid URI. Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com> --------- Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
#2520 introduced v2 facets in Python client. Rest of Python code requires migration to use them.
Solution
Remove usage of v1 facets. In apache/airflow#39614 there is removed
openlineage.commondependency on BigQuery in Google provider so there should be no discrepancy between versions.One-line summary:
Migrate integrations to use v2 python facets.
Checklist
SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project