Fix unhashable issue with secrets.backend_kwargs and caching #25970
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pdebelak
force-pushed
the
fix-secrets-manager-dict-arguments
branch
from
267e37a
to
8c620fe
Compare
uranusjr
approved these changes
Aug 26, 2022
|
I like this aproach :). Thanks for quick fixing @pdebelak :) |
anja-istenic
pushed a commit
to anja-istenic/airflow
that referenced
this pull request
Aug 29, 2022
Also a chapter was added to recommend taking a backup before the migration. Based on discussions and user input from apache#25866, apache#24526 Closes: apache#24526 Improve cleanup of temporary files in CI (apache#25957) After recent change in Paralell execution, we start to have infrequent "no space left on device" message - likely caused by the /tmp/ generated files clogging the filesystem from multiple runs. We could fix it by simply running cleanup after parallel job always, but this is not good due to diagnostics needed when debugging parallel runs locally so we need to have a way to skip /tmp files deletion. This PR fixes the problem twofold: * cleanup breeze instructions which is run at the beginning of every job cleans also /tmp file * the parallel jobs cleans after themselvs unless skipped. Properly check the existence of missing mapped TIs (apache#25788) The previous implementation of missing indexes was not correct. Missing indexes were being checked every time that `task_instance_scheduling_decision` was called. The missing tasks should only be revised after expanding of last resort for mapped tasks have been done. If we find that a task is in schedulable state and has already been expanded, we revise its indexes and ensure they are complete. Missing indexes are marked as removed. This implementation allows the revision to be done in one place Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> Fix dataset_event_manager resolution (apache#25943) Appears `__init__` is not invoked as part of `_run_raw_task` due to the way TI is refreshed from db. Centralize dataset manager instantiation instead. Fix unhashable issue with secrets.backend_kwargs and caching (apache#25970) Resolves apache#25968 Fix response schema for list-mapped-task-instance (apache#25965) update areActiveRuns, fix states (apache#25962)
pdebelak
added a commit
to pdebelak/airflow
that referenced
this pull request
Sep 7, 2022
…pache#25970)" This reverts commit aa87763.
potiuk
pushed a commit
that referenced
this pull request
Sep 8, 2022
Revert "Fix unhashable issue with secrets.backend_kwargs and caching (#25970)"
jedcunningham
added
the
changelog:skip
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This alters the caching for secrets backend so it can handle dictionary or list arguments. It does so by caching on the backend_kwargs as a json string before trying to parse the json.
closes: #25968
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.