KEP-5901: Add Kubectl Checkpoint KEP #5092
Conversation
k8s-ci-robot
added
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: adrianreber The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
needs-ok-to-test
|
Hi @adrianreber. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/XXL
| Beta in Kubernetes 1.30, which means that the corresponding feature gate | ||
| defaults to the feature being enabled, the next step would be to extend the | ||
| existing checkpointing functionality from the *kubelet* to *kubectl* for easier | ||
| user consumption. The main motivation is to make it easier by not requiring |
There was a problem hiding this comment.
I'm not familiar with the criteria followed for kubectl commands, but should not wait first for the feature to be GA so it is available? or is the plan to GA KEP-2008 and add the kubectl command at the same time?
There was a problem hiding this comment.
I also don't know. Not sure. But as there is a possibility to add plugins for kubectl or alpha commands I thought it is possible to expose non GA features on the api server level. But I don't know.
| Currently the design details are based on the existing pull request: [Add | ||
| 'checkpoint' command to kubectl][pr120898] | ||
|
|
||
| The API server is extended to handle checkpoint requests from *kubectl*: |
There was a problem hiding this comment.
This is the most important change , adding a new endpoint to the apiserver is where you need to expand, Jordan also commented here kubernetes/kubernetes#120898 (comment) along those lines
There was a problem hiding this comment.
I am sorry. But what is necessary. The comment you linked to was talking about the kubelet KEP which didn't had any reference to kubectl or the to the API server. This here is created to address the mentioned comment.
| for the initialization to finish. The startup time is reduced to the time | ||
| necessary to read back all memory pages to their previous location. | ||
|
|
||
| This feature is already used in production to decrease startup time of |
There was a problem hiding this comment.
claims should have links to references
| This feature is already used in production to decrease startup time of | ||
| containers. | ||
|
|
||
| Another similar use case for quicker starting containers has been reported in |
| #### Optimize Resource Utilization | ||
|
|
||
| This use case is motivated by interactive long running containers. One very | ||
| common problem with things like Jupyter notebooks or remote development |
There was a problem hiding this comment.
naive questions, is this state not stored in a database or some persistent storage and recovered when it reconnects? at the end you'll have to keep all the state stored somewhere
|
|
||
| #### Container Migration | ||
|
|
||
| One of the main use cases for checkpointing and restoring containers is |
There was a problem hiding this comment.
migration between nodes? the IPs are most likely to be lost so the application has to be agnostic of the IP per example
There was a problem hiding this comment.
Added a paragraph concerning migration of TCP connections.
| migrate containers or processes. It is a well researched topic especially | ||
| in the field of high performance computing (HPC). To avoid loss of work | ||
| already done by a container the container is migrated to another node before | ||
| the current node crashes. There are many scientific papers describing how |
There was a problem hiding this comment.
one or two references to this papers will be nice
| case, only useful for stateful containers. | ||
|
|
||
| With GPUs becoming a costly commodity, there is an opportunity to help | ||
| users save on costs by leveraging container checkpointing to prevent |
There was a problem hiding this comment.
Workloads are already doing checkpointing , do you know what is the state of the art of existing checkpointing mechanisms vs container checkpointing?
There was a problem hiding this comment.
That is the hard question for the last 25 years. What is better. Application level checkpointing or system level checkpointing. Both approaches have their advantages and drawbacks. As it is unlikely that every application will have application level checkpointing some workloads can only be migrated with system level checkpointing.
Currently there are multiple startups and scientific researchers trying to solve how to better use GPU resources. All of the one I have been following are betting on system level checkpointing as application level checkpointing does not work or exist. The main problem is that for every application it has to be re-implemented. But as I am coming from the system level checkpointing area I am probably biased.
| Container migration for load balancing is something where checkpoint/restore | ||
| as implemented by CRIU is already used in production today. A prominent example | ||
| is Google as presented at the Linux Plumbers conference in 2018: | ||
| [Task Migration at Scale Using CRIU]<[task-migration]> |
There was a problem hiding this comment.
The example says that connections are dropped and client must reconnect, this is well understood at google where are librarie and applications that handle the client side reconnection, but my observation is that most people expect to auto-magically reconnect, and AFAIK this will not do it
There was a problem hiding this comment.
Added a paragraph that talks about TCP connection and checkpoint/restore.
| ##### Spot Instances | ||
|
|
||
| Yet another possible use case where checkpoint/restore is already used today | ||
| are spot instances. Spot instances are usually resources that are cheaper but |
There was a problem hiding this comment.
This will need to take into account the time you have for checkpointing, as spot is like that, eventually you'll get destroyed
There was a problem hiding this comment.
Added a reference to an existing solution that handles this.
| Also, *kubectl* is extended to call this new API server interface. The API | ||
| server, upon receiving a request, will call the kubelet with the corresponding | ||
| parameters passed from *kubectl*. Once the checkpoint has been successfully written | ||
| to disk *kubectl* will return the name of the node as well as the location of | ||
| the checkpoint archive to the user: |
There was a problem hiding this comment.
as commented above, this is the most tricky part of the KEP, you need to expand on the technical design here, these endpoints are complex to implement also you need to play with version skews between apiserver, kubelet and container runtimes
There was a problem hiding this comment.
Unfortunately I am not sure what is needed here. I described the API to be just as the API provided by the kubelet. It just forwards everything 1:1 to the kubelet. Concerning different versions of api server and kubelet I described in one section that it will probably just return an error. I guess I do not get it what is required here.
Any existing examples I can take a look at?
With "Forensic Container Checkpointing" being Beta and discussions around graduating it to GA, the next step would be kubectl integration of the container checkpointing functionality. In addition to the "Forensic Container Checkpointing" use case this KEP lists multiple use cases how checkpointing containers can be used. One of the main motivations for this KEP is to make it easier for users to checkpoint containers, independent of the reason. Having it available via kubectl reduces the complexity of connecting to the node and accessing the kubectl checkpoint API endpoint. Signed-off-by: Adrian Reber <areber@redhat.com>
adrianreber
force-pushed
the
2025-01-27-kubectl-checkpoint
branch
from
1b65fed to
cccb3cb
Compare
|
@aojea thanks for your review. I added a couple more references. |
With "Forensic Container Checkpointing" being Beta and discussions around graduating it to GA, the next step would be kubectl integration of the container checkpointing functionality.
In addition to the "Forensic Container Checkpointing" use case this KEP lists multiple use cases how checkpointing containers can be used.
One of the main motivations for this KEP is to make it easier for users to checkpoint containers, independent of the reason. Having it available via kubectl reduces the complexity of connecting to the node and accessing the kubectl checkpoint API endpoint.