Sync v2-2-stable with v2-2-test to release 2.2.3
#20206
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This will skip the auto-generated file from licence check.
(cherry picked from commit 2bafc08)
(cherry picked from commit cb6137f)
(cherry picked from commit 9828773)
(cherry picked from commit cc627b3)
(cherry picked from commit 64b8389)
) (cherry picked from commit 4996501)
We need to use the job_id used to queue the TI, not the current schedulers job_id. These can differ naturally with HA schedulers and with scheduler restarts (clearing "queued but not launched TIs" happens before adoption). (cherry picked from commit b80084a)
Airflow regularly reloads sys.modules, which makes type identity comparison unreliable, because a class would obtain a second, different identity in the interpreter when imported after a reload. This makes validation difficult because there isn't really a way to tell whether two class objects are indeed "the same". But this check is only for sanity to begin with, so the best we can do is to drop the check entirely ans trust the Plugin Manager is doing its job correctly. (cherry picked from commit 7d555d7)
(cherry picked from commit 0591852)
Removes extra noise and actually mounts ``airflow.cfg``. (cherry picked from commit 53b2415)
(cherry picked from commit 264cb09)
add missing ')' in code example (cherry picked from commit f8a6cbb)
(cherry picked from commit c167050)
The DagFileProcessor.manage_slas does not consider if an SlaMiss already exists in DB while inserting slas. If an SLA for a task is missed and recorded, on checking SLA again, this task comes up again if there's no recent run of the task and we try to insert the record into the SlaMiss table again, this results in Integrity error. This PR fixes that by avoiding insert if the record already exists Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> Co-authored-by: Kaxil Naik <kaxilnaik@apache.org> (cherry picked from commit 9519bf6)
) * Fix task instance api cannot list task instances with None state The task instance state can be None and in the API we accept `none` for null state. This PR fixes this issue by converting the `none` to None and improving the query so that the DB can get this state. (cherry picked from commit f636060)
Currently, the user must specify image and tag in airflow.cfg, even when they are using a pod template file. If the pod template file specifies an image and tag, the user should not be forced to also specify this in airflow.cfg. (cherry picked from commit d18e2b0)
(cherry picked from commit e99c14e)
An active dag can suddenly have import errors as a result of the DAG file being changed. Currently, we do not consider this before creating dagruns. This PR adds the has_import_errors to dagmodel so dags with import errors are not sent to the scheduler to create dagruns Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> (cherry picked from commit 3ccb794)
There were many questions recently along the line of "How do I access context from TaskFlow task". Surprisingly, the paragraph about accessing current context was removed from the "Concepts" (where it was there for Airflow 2.0.0) but was never added to the "TaskFlow Tutorial" where it actually belongs. Also Python Operator's description about passing context variables as kwargs have been removed when `provide_context` parameter was removed (it was only present in the docstring of `provide_context` and you could likely deduce this behaviour from several examples, but it was not mentioned anywhere. This PR adds the description with examples to the Python operator as well as adds similar description in TaskFlow tutorial, including the possibility of using `get_current_context` deep down the stack to retrieve the context variables even if they are not passed via kwargs. (cherry picked from commit d6c8730)
(cherry picked from commit b4321de)
Remove "the" from this line: "Airflow defines the some Jinja filters that can be used to format values." (cherry picked from commit 0131726)
Fix typo (cherry picked from commit a256af0)
One of the challenges for Airflow users it to understand how the Airflow Scheduler turns Python code into DAGs. Often users will be confused as to why their DAGs don't show up in the user interface, or they'll see DAG processing errors and not know why. This documentation entry is intended to clarify the process to new and experienced users alike, by providing a concise description of the process, along with a diagram of the flow and links to key configuration values that affect the way Airflow processes DAGs. (cherry picked from commit d119fee)
(cherry picked from commit 3f6d9b6)
(cherry picked from commit 5061f52)
(cherry picked from commit ee8e0b3)
Co-authored-by: Tzu-ping Chung <tp@astronomer.io> (cherry picked from commit ba6b7c7)
* fixing #19028 by having chown be in a sudo call * removing unused import * trying to clean up a test * combine sudo chown calls * force exception when chown fails * Update tests/task/task_runner/test_base_task_runner.py * Fix tests * Fix formatting Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> Co-authored-by: Ash Berlin-Taylor <ash@apache.org> (cherry picked from commit b37c0ef)
(cherry picked from commit f59decd)
Co-authored-by: Bas Harenslak <bas@astronomer.io> (cherry picked from commit 01c7c4d)
(cherry picked from commit 1c2dfde)
The language "when two tasks defer based on the same trigger" is a bit confusing. Many tasks can reuse the same trigger class. But two tasks can't defer using the same trigger _instance_. I think what's important to call out here, and what I try to make clearer, is that the exact same instance of the trigger may have multiple copies of itself running. Additionally I clarify cleanup is not _only_ called "when this happens" (that is, when trigger is "suddenly removed"), but called every time the trigger instance exits, no matter the reason. (cherry picked from commit 9a92a56)
Related to #18453 (comment) `1.5.0` was yanked so `>=1.5.1` is safe and we already have `1.7.5` in constraints-main (cherry picked from commit f99d0e7)
(cherry picked from commit 82102e6)
jedcunningham
requested review from
XD-DENG,
ashb,
bbovenzi,
dimberman,
ephraimbuddy,
kaxil,
mik-laj,
potiuk and
ryanahamilton
as code owners
boring-cyborg
bot
added
area:API
kaxil
reviewed
Dec 10, 2021
kaxil
approved these changes
Dec 10, 2021
github-actions
bot
added
the
full tests needed
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
jedcunningham
force-pushed
the
v2-2-test
branch
from
54cc40d to
db8d78f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Time for
2.2.3rc1!