Use kubernetes queue in kubernetes hybrid executors #23048
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tanelk
requested review from
dstandish,
jedcunningham,
kaxil,
XD-DENG and
ashb
as code owners
boring-cyborg
bot
added
provider:cncf-kubernetes
2 tasks
|
Pinging @ashb for a review of this bugfix. |
|
@uranusjr, could you take a look at this bugfix |
|
any update on this guys? |
|
this can help us thanks, eagerly waiting for it 👍 |
ashb
approved these changes
May 4, 2022
|
@tanelk Could you rebase please? |
github-actions
bot
added
the
full tests needed
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
2 tasks
|
@snjypl not "instead of", but more likely "additionally". The queue filter is needed for this particular bug because every job can queue celery and kubernetes tasks. |
ephraimbuddy
pushed a commit
that referenced
this pull request
May 8, 2022
When using "hybrid" executors (`CeleryKubernetesExecutor` or `LocalKubernetesExecutor`), then the `clear_not_launched_queued_tasks` mechnism in the `KubernetesExecutor` can reset the queued tasks, that were given to the other executor. `KuberneterExecutor` should limit itself to the configured queue when working in the "hybrid" mode. (cherry picked from commit ae19eab)
ephraimbuddy
pushed a commit
that referenced
this pull request
May 21, 2022
When using "hybrid" executors (`CeleryKubernetesExecutor` or `LocalKubernetesExecutor`), then the `clear_not_launched_queued_tasks` mechnism in the `KubernetesExecutor` can reset the queued tasks, that were given to the other executor. `KuberneterExecutor` should limit itself to the configured queue when working in the "hybrid" mode. (cherry picked from commit ae19eab)
61 tasks
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When using "hybrid" executors (
CeleryKubernetesExecutororLocalKubernetesExecutor), then theclear_not_launched_queued_tasksmechnism in theKubernetesExecutorcan reset the queued tasks, that were given to the other executor.KuberneterExecutorshould limit itself to the configured queue when working in the "hybrid" mode.This is reported at least once: #21225 (comment) , but has not been dealt with.
Fixes #22554
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.