Immediately fail the task in case of worker pod having a fatal container state #37670
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:providers
provider:cncf-kubernetes
potiuk
reviewed
Feb 27, 2024
|
But also would love to get someone else to take a look |
hussein-awala
approved these changes
Mar 1, 2024
There was a problem hiding this comment.
This is a nice improvement, I was worried about the sub-reason for each reason (eg: pull QPS exceeded which I'm not sure if it's the exact thing in all the K8S versions and distributions), but as the user has the option to update the reasons list, he can fix it without upgrading the provider version if we detect another similar case or any bug.
LGTM
79 tasks
abhishekbhakat
pushed a commit
to abhishekbhakat/my_airflow
that referenced
this pull request
Mar 5, 2024
…ner state (apache#37670) * fail the task in case of worker pod having fatal container state * version number updated
utkarsharma2
pushed a commit
to astronomer/airflow
that referenced
this pull request
Apr 22, 2024
…ner state (apache#37670) * fail the task in case of worker pod having fatal container state * version number updated
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What happened
When the worker pods init/base containers are in a pending state due to fatal container
state reasons, the tasks eventually fail and the pods are deleted. Currently, it has to wait until the worker_pods_pending_timeout even though the worker pods don't recover.
What do you think should happen instead
When the worker pods init/base containers are in a pending state due to fatal container
state reasons, the worker pod doesn't recover. It doesn't make sense to wait until the worker_pods_pending_timeout. Instead, mark the tasks as failed and delete the worker pods.