Release 1.4.2 #1028
Merged
Release 1.4.2 #1028
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix three problems observed while running the tests in the CI recently: 1. Static check Fix spelling issue captured by static check (#1000) 2. Integration test that's no longer needed The issue we were trying to capture for no longer happens in the latest version of the Apache Airflow provider `apache-airflow-providers-postgres==5.11.1rc1`: apache/airflow#39842 3. Skip a buggy version of OL There was a breaking change between `openlineage-integration-common==1.14.0` and `openlineage-integration-common==1.15.0` . It may have been an unintended side-effect of OpenLineage/OpenLineage#2693. This is an example of how Cosmos was using `DbtLocalArtifactProcessor` - something that had been agreed upon in the past: ``` openlineage_processor = DbtLocalArtifactProcessor( producer=OPENLINEAGE_PRODUCER, job_namespace=LINEAGE_NAMESPACE, project_dir=project_dir, profile_name=self.profile_config.profile_name, target=self.profile_config.target_name, ) events = openlineage_processor.parse() for completed in events.completes: for output in getattr(completed, source): dataset_uri = output.namespace + "/" + output.name uris.append(dataset_uri) ``` In `openlineage-integration-common==1.14.0` and earlier versions, this would create URIs in the format: ``` postgres://0.0.0.0:5432/postgres.public.stg_customers ``` Since openlineage-integration-common==1.15.0 , this leads to URIs being created in the format: ``` postgres.public.stg_customers/postgres://0.0.0.0:5432 ``` This was fixed in OpenLineage/OpenLineage#2735 and released as part of OL 1.16: https://github.com/OpenLineage/OpenLineage/releases/tag/1.16.0
<!--pre-commit.ci start--> updates: - [github.com/codespell-project/codespell: v2.2.6 → v2.3.0](codespell-project/codespell@v2.2.6...v2.3.0) - [github.com/astral-sh/ruff-pre-commit: v0.4.4 → v0.4.5](astral-sh/ruff-pre-commit@v0.4.4...v0.4.5) <!--pre-commit.ci end--> Additionally, given the latest `apache-airflow-providers-postgres==5.11.1rc1`, mentioned in apache/airflow#39842, it no longer validates the Dataset URI as it used to - so the test case we had in Cosmos is no longer relevant. Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
## Fix dev Dockerfile The Dockerfile was not building because `dbt-postgres==1.8.0` pinned `psycopg2`. To resolve this, you need to apt-get install `build-essential` and `libpq-dev`. I added a `RUN` directive that does that. This is placed near the top as it makes the most sense for layering purposes. ## Add uv While I was at it, I added uv: https://github.com/astral-sh/uv tldr: this makes the Dockerfile build a little faster. This speed is especially important due to how the Dockerfile will typically be used: if a user makes a change to the `cosmos/` directory and then wants to test their code, due to how the layers are stacked, the pip install will need to be re-run and resolved on over 100 Python packages on each code change.
We currently do not have GitHub issue templates when the community creates issues. Having issue templates would aid us in catching relevant information that would help in faster resolution of those issues reported. With this PR, we're are adding issue templates for bug reports and feature requests. closes: #977
Improve error logging when the `dbt` command returns a non-zero exit code. Instead of raising an `AirflowException` with the full output, log the output using the logger and then raise the exception with a concise error message. This makes the dbt output more readable and not in a single line as AirflowException logs message in a single line, and it can get very long. closes #1003 --------- Co-authored-by: David Steinar Asgrimsson <david-steinar.asgrimsson@kommuninvest.se> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
<!--pre-commit.ci start--> updates: - [github.com/astral-sh/ruff-pre-commit: v0.4.5 → v0.4.7](astral-sh/ruff-pre-commit@v0.4.5...v0.4.7) <!--pre-commit.ci end--> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…1023) - To allow setting the invocation mode to `InvocationMode.SUBPROCESS` when using `ExecutionMode.VIRTUALENV`. This should unblock - Automatically setting the invocation mode to `InvocationMode.SUBPROCESS` if user does not specify the invocation mode when using `ExecutionMode.VIRTUALENV` because it is the only supported invocation mode. Closes: 998
## Description This PR enable CI to run unit tests for Python3.12 CI Job: https://github.com/astronomer/astronomer-cosmos/actions/runs/9357004387/job/25755741232?pr=1018 ## Related Issue(s) closes: #964 <!-- If this PR closes an issue, you can use a keyword to auto-close. --> <!-- i.e. "closes #0000" --> ## Breaking Change? <!-- If this introduces a breaking change, specify that here. --> ## Checklist - [ ] I have made corresponding changes to the documentation (if required) - [ ] I have added tests that prove my fix is effective or that my feature works
Include the following in the dbt & Airflow dependencies compatibility matrix - dbt 1.8 - Airflow 2.9
As of Cosmos 1.4.1, users are not able to disable the cache in Cosmos using `AIRFLOW__COSMOS__ENABLE_CACHE=0`. During a recent refactoring #975, the `enable_cache` was changed to a non-boolean config. This PR fixes this issue.
#1017) This PR follows the methodology in #683 by modifying the GCP `GoogleCloudServiceAccountDictProfileMapping()` `profile_args` not require a dataset as a required argument. DAG RUN <img width="1473" alt="Screenshot 2024-06-05 at 6 50 04 PM" src="https://github.com/astronomer/astronomer-cosmos/assets/98807258/81b127b9-c5e5-4983-9efe-bbf00d81914f"> Co-authored-by: "Ollie Ma" <oliver.zheyi.ma@gmail.com> Original PR by @oliverrmaa: #839 Closes #837
dosubot
bot
added
the
size:XL
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
dosubot
bot
added
area:ci
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1028 +/- ##
==========================================
- Coverage 95.67% 95.65% -0.03%
==========================================
Files 59 59
Lines 2890 2899 +9
==========================================
+ Hits 2765 2773 +8
- Misses 125 126 +1 ☔ View full report in Codecov by Sentry. |
pankajkoti
approved these changes
Jun 6, 2024
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
Bug fixes * Fix the invocation mode for ``ExecutionMode.VIRTUALENV`` by @marco9663 in astronomer#1023 * Fix Cosmos ``enable_cache`` setting by @tatiana in astronomer#1025 * Make ``GoogleCloudServiceAccountDictProfileMapping`` dataset profile arg optional by @oliverrmaa and @pankajastro in astronomer#839 and astronomer#1017 * Athena profile mapping set ``aws_session_token`` in profile only if it exists by @pankajastro in astronomer#1022 Others * Update dbt and Airflow conflicts matrix by @tatiana in astronomer#1026 * Enable Python 3.12 unittest by @pankajastro in astronomer#1018 * Improve error logging in ``DbtLocalBaseOperator`` by @davidsteinar in astronomer#1004 * Add GitHub issue templates for bug reports and feature request by @pankajkoti in astronomer#1009 * Add more fields in bug template to reduce turnaround in issue triaging by @pankajkoti in astronomer#1027 * Fix ``dev/Dockerfile`` + Add ``uv pip install`` for faster build time by @dwreeves in astronomer#997 * Drop support for Airflow 2.3 by @pankajkoti in astronomer#994 * Update Astro Runtime image by @RNHTTR in astronomer#988 and astronomer#989 * Enable ruff F linting by @pankajastro in astronomer#985 * Move Cosmos Airflow configuration to settings.py by @pankajastro in astronomer#975 * Fix CI Issues by @tatiana in astronomer#1005 * Pre-commit hook updates in astronomer#1000, astronomer#1019 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Reeves <31971762+dwreeves@users.noreply.github.com> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: davidsteinar <davidsteinar@gmail.com> Co-authored-by: David Steinar Asgrimsson <david-steinar.asgrimsson@kommuninvest.se> Co-authored-by: Marco Yuen <33394715+marco9663@users.noreply.github.com> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> Co-authored-by: Ollie Ma <oliver.zheyi.ma@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bug fixes
ExecutionMode.VIRTUALENVby @marco9663 in Allow setting invocation mode when usingExecutionMode.VIRTUALENV#1023enable_cachesetting by @tatiana in Fix Cosmos enable_cache setting #1025GoogleCloudServiceAccountDictProfileMappingdataset profile arg optional by @oliverrmaa and @pankajastro in Make GoogleCloudServiceAccountDictProfileMapping dataset profile argument optional #839 and Make GoogleCloudServiceAccountDictProfileMapping dataset profile argu… #1017aws_session_tokenin profile only if it exists by @pankajastro in Athena profile mapping set aws_session_token in profile only if it exist #1022Others
DbtLocalBaseOperatorby @davidsteinar in Improve error logging in DbtLocalBaseOperator #1004dev/Dockerfile+ Adduv pip installfor faster build time by @dwreeves in Fixdev/Dockerfile+ Adduv pip installfor faster build time #997