-
Notifications
You must be signed in to change notification settings - Fork 39.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Waiting for Job completion in InitContainer (or before main container starts) #106802
Comments
@collimarco: This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/sig apps node |
The current workaround solution is Fine Parallel Processing Using a Work Queue. If using init container, https://github.com/groundnuty/k8s-wait-for is a choice to wait for the job completion. Not sure if I misunderstand. The expected solution here is adding
|
This sounds rather like the opposite. To hold the start of a Deployment until an existing Job finishes. I think you could use a pipeline framework for this, like kubeflow-pipelines or argo. I don't think the need is generic enough to fit in k8s. |
Actually most of the examples for Also consider that Heroku has "Release Phase", meaning that migrations between releases are a common need: it seems something that Deployment already has the concept of multiple versions of the app running at the same time, so it would be interesting to have something to facilitate the transition between the two app versions (i.e. reference a migration |
Have you looked into kubeflow-pipelines, Argo or Tekton? Let me rephrase: I think this need can be done in the application level, rather than being part of core kubernetes. There are plenty of frameworks (listed above) which provide the functionality. |
@alculquicondor I have been searching for months for a clean solution (do you have any useful links?) and the best solution that I have found is to run a migration The solution with Also a |
Note that waiting in a initContainer is potentially wasteful (the nodes are allocated with the pod resources, but nothing is actually running on them). I think the ideal solution is that the Deployment is not created until the Job finishes. You could achieve this from the Job itself (by using the k8s API to create the deployment). But the frameworks I already mentioned focus on this exact problem. Probably Argo fits this use case better https://argoproj.github.io/ |
@alculquicondor It doesn't seem like a waste waiting for a migration to complete for a few minutes... The pods would be there in any case for running the application. Yes, I know about Argo, I meant if you have a specific link for database migrations. As I already said I made extensive research on the topic and it seems that most Rails applications use the strategy that I described above. |
I think an Argo workflow solves the problem elegantly, from an API point of view. I'm not in the domain of database migrations, so I don't know about people doing this exact thing with Argo. But my point is that there seems to be a solution and, thus, not enough justification to bake this into the Deployment/ReplicaSet API. But, I'm not an SIG Apps approver. If you feel strongly about it, you can join the SIG meetings to ask for feedback. |
From the official K8s documentation:
It seems exactly my use case: I'm using them to wait for a condition (migration completed).
Ok, can you tag someone of your team with this specific knowledge? |
@kow3ns ? |
@collimarco |
@denkensk Yes, that is a normal Job defined in Argo Workflows... However you are defining a sequence of imperative commands and you can't use an entirely declarative approach. On the other hand, using For example one of the drawbacks of Argo Workflows approach is that you don't have the updated environment because you run the migration before the deploy. On the other hand, using |
@collimarco In my personal opinion, I think the But based on my actual experience with Argo-Workflow in a production environment, the graphics may be more complex, there will be at least 5 or more stages. If we chose to use InitContainer to do this, we will face some issues below:
|
Pods are replaced by rolling updates managed by Deployments, so, using my strategy, you don't have more pods and you don't have resource problems... It's just like a normal rolling update.
Many apps can run on a simple PaaS like Heroku and they don't need so many stages... For example Heroku has only 1 stage/command in the release phase. It would be interesting to have a strategy inside Kubernetes - which is becoming the standard - for this common use case. |
I suggest you join a SIG Apps meeting to discuss your suggestion. |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
Did you ever bring this up to the SIG Apps meeting? Also @imjasonh, this kind of sounds similar to what you were talking about at WG Batch |
any update? |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
I would like to respond to your recent comments, which I find inappropriate and incorrect. I firmly believe that we need this functionality directly within Kubernetes, rather than relying on external tools like ArgoCD. It is evident that your lack of understanding and unwillingness to acknowledge the need for this functionality demonstrates a probable lack of practical experience in a production environment. The responses from other users clearly confirm this need, yet you persistently ask "why?" without grasping the concept. DevOps engineers with extensive experience in the world of Kubernetes understand the challenges associated with waiting for the completion of dependent tasks or migrations. They recognize the importance of having this functionality directly within the core of Kubernetes. Your rejection of this idea and attempts to redirect us to other tools are out of touch with reality and do not support progress and the simplicity of application management on Kubernetes. Implementing this feature would provide significant added value and streamline the work of many teams dealing with the complexities of dependency management and migrations. It is crucial to listen to the needs of the community and add features that are genuinely useful and necessary. Furthermore, other users have already come up with various ways to address this problem, which demonstrates a genuine demand for a native feature to wait for task completion in Kubernetes. I assume that your inquiries regarding this topic stem from a desire to improve Kubernetes and contribute to its further development. Therefore, I kindly request that you reconsider your stance, take into account the feedback from other users, and consider the benefits this functionality would bring to the broader community. Thank you for your understanding. |
This is an open source project. Everyone is welcome to come with a proposal, share, discuss it, etc. Sometimes it's not enough to say "I need this feature". What are the complete semantics of the feature? Why are alternatives not enough? This is all encompassed in the KEP process: https://github.com/kubernetes-sigs/kueue/tree/main/keps/NNNN-template As I have already stated it: if you believe this is a needed feature, share a proposal, come to a SIG meeting, and we'll guide you through the process. |
@alculquicondor Here is the KEP Draft based on the provided template. Can you make a review? I'm also very busy (and also I'm more DevOps than Developer) so I cannot contribute code, but hopefully ... I found some spare time to prepare this draft (it's my first KEP so it will need more work anyway). Maybe other guys from this thread (e.g. @collimarco, @iocanel, @Tensho, @rahul-sharma-78) will help to move it further 😉: KEP-NNNN: Waiting for required Pod(s) to be Running/Succeeded before dependent Pod startsSummaryThis KEP proposes the addition of a new functionality in Kubernetes that allows pods to wait for specific pods to reach the Running or Succeeded state before starting. This feature is crucial for ensuring proper sequencing and dependencies between pods in a Kubernetes cluster. MotivationIn certain scenarios, a pod must wait for one or more prerequisite pods to be in the Running or Succeeded state before it can start. This functionality is currently missing in Kubernetes, leading to challenges in managing dependencies and sequencing of pods. This KEP addresses this limitation and provides a robust mechanism for handling dependencies between pods. Goals
Non-Goals
ProposalThe proposed solution involves adding a new field to the pod specification that allows users to specify the pods on which a dependent pod should wait. This field can accept multiple pod selectors or pod names, allowing for flexible configuration. User Stories (Optional)Story 1As a Kubernetes user, I want to deploy a pod that relies on the successful completion of a specific job pod. I should be able to define the dependency between these pods so that my dependent pod starts only when the job pod has been successfully completed. Notes/Constraints/Caveats (Optional)
Risks and Mitigations
Design DetailsImplementation ProposalThe proposed implementation involves modifying the Kubernetes scheduler and pod startup logic to accommodate the waiting functionality. When a dependent pod is scheduled, the scheduler checks the specified prerequisite pods' status and waits until they reach the Running or Succeeded state before starting the dependent pod. Test PlanPrerequisite testing updatesThe existing unit tests and integration tests for pod scheduling and startup logic need to be updated to cover the new waiting functionality. Unit Tests
Integration Tests
Graduation CriteriaThe proposed functionality will go through the Kubernetes enhancement process and will be considered graduated when it has been implemented, tested, and reviewed by the Kubernetes community. Implementation History
Drawbacks
AlternativesDelay job start instead of workload admissionInstead of delaying the admission of dependent workloads, an alternative approach could be to delay the start of the job itself until all prerequisite pods are in the required state. Pod Resource ReservationAnother alternative is to introduce a pod resource reservation mechanism, where pods can reserve resources and wait for them to become available before starting. This approach may require changes to the Kubernetes scheduler and resource allocation logic. More granular configuration to enable the mechanismInstead of a single field for specifying prerequisite pods, a more granular configuration option could be provided to enable the waiting mechanism. This would allow users to define custom conditions and dependencies based on their specific requirements. |
/remove-lifecycle stale |
FYI @JuryA you should post your KEP under https://github.com/kubernetes/enhancements, it's more convenient for discussion, you can follow the guide here https://github.com/kubernetes/enhancements#when-to-create-a-new-enhancement-issue |
For anyone looking to proceed with a KEP, I would like to see the following questions answered:
Also, it looks like you abandoned the idea of waiting in an init container. Is this correct? This is probably good, but worth listing in alternatives. |
Thank you for your guidance. I will incorporate the missing points and relocate it to the proposed location. |
It would be great to see this KEP posted and implemented, I'd definitely show support for such feature. |
@JuryA did you get that posted to the kubernetes/enhancements project? Do you have a direct link? I'd really like to follow this, as one more person who would really like to see this done. @alculquicondor I don't think anyone is arguing that the InitContainer approach is what we want... we're saying it's the ugly alternative we have to live with because there's nothing native. I'd much rather see it not even schedule the pod until the precondition is met (much like it does when the pod needs a PVC). Your list of question is valid and all should be ironed out in the KEP, but, as several people have pointed out, this is a near-ubiquitous need, and not having answers to all the questions doesn't mean the request isn't valid. As someone who has a CI pipeline built outside argo, being told k8s doesn't support my extremely common use case unless I migrate my entire CI pipeline to a third-party offering really makes k8s look like an incomplete solution. |
I didn't say it's not.
You need to think of k8s as an operating system kernel. Hence why one of the questions is how we can make writing workflow managers easier, instead of implementing a full blown workflow manager. |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
Hi, |
Can we get the link to the KEP posted here please? |
Did anyone make the KEP for this? Anyone have a link? @JuryA ? |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close not-planned |
@k8s-triage-robot: Closing this issue, marking it as "Not Planned". In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
What would you like to be added?
A common pattern is to include the release phase (e.g. database migrations) inside a Job.
Then you need to wait until the migration is completed before starting the new pods: you can use an InitContainer for that.
For example if you use Rails you can:
migration-job
) withrails db:migrate
commandrake db:abort_if_pending_migrations
(this command returns success only when the migrations are complete)The problem is that not all commands (in step 1) have a corresponding "check" command (in step 2). It can be extremely hard for some commands to find a separate command in step 2 that verifies their completion.
It would be much easier if we could simply wait for a Job to become Complete before starting the pods (in step 2).
Why is this needed?
InitContainers are a great way to wait for a general condition to be met, but it is difficult to check a condition for a Kubernetes component (like a Job).
One of the most common conditions is a Job completion, but currently there's no way to express that in Kubernetes (e.g.
waitFor: migration-job
). This could be something similar to whatkubectl wait
does on the client side, but inside the spec.The text was updated successfully, but these errors were encountered: