Add experimental CRI API for checkpoint/restore #97689

adrianreber · 2021-01-04T17:02:39Z

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

This PR contains the first two commits of #97194. I created this PR to make it easier to review the CRI API changes without all the changes for the actual feature implementation in #97194.

I also want to make it easier to continue the discussion in kubernetes/enhancements#1990

This introduces a new, experimental CRI API.

Which issue(s) this PR fixes:
It partly fixes #3949 but it is just a step towards solving #3949 . The next step would be #97194 .

Special notes for your reviewer:
Just two commits split out of #97194 for easier review.

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
kubernetes/enhancements#1990

k8s-ci-robot · 2021-01-04T17:02:47Z

@adrianreber: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2021-01-04T17:02:48Z

Hi @adrianreber. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

fejta-bot · 2021-01-04T17:32:50Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

dims · 2021-01-04T18:58:17Z

/assign @derekwaynecarr @mrunalp

Derek, Mrunal, how does this fit into the CRI API stuff in progress?

fejta-bot · 2021-01-04T21:09:50Z

Unknown CLA label state. Rechecking for CLA labels.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/check-cla

krmayankk · 2021-01-05T08:46:32Z

staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto

+    // CheckpointContainer checkpoints a container
+    rpc CheckpointContainer(CheckpointContainerRequest) returns (CheckpointContainerResponse) {}
+    // RestoreContainer restores a container
+    rpc RestoreContainer(RestoreContainerRequest) returns (RestoreContainerResponse) {}


does this apply to all types of containers ? e.g. init and sidecars flagged as well?

does this apply to all types of containers ? e.g. init and sidecars flagged as well?

In my drain --checkpoint implementation (#97194) I am ignoring containers in the 'kube-system' namespace. As long as no raw sockets are used CRIU could checkpoint most processes and containers, but it does not seem useful to checkpoint containers in the 'kube-system' namespace for now.

Checkpointing a Pod in CRI-O also does not checkpoint the pause container, only necessary metadata is written to the checkpoint archive and upon restore a new pause container is created which uses the metadata to provide the same environment for the other restored containers.

Thanks @adrianreber , i was asking about https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
Also does it make sense for the CRI to know about kubernetes namespace kube-system at the CRI level ? I believe that logic should be built into kubelet and not into CRI

i was asking about https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

This was briefly discussed in the corresponding KEP and one idea was to not enable checkpointing before init containers have finished running.

Also does it make sense for the CRI to know about kubernetes namespace kube-system at the CRI level ? I believe that logic should be built into kubelet and not into CRI

Currently it happens in kubectl. Only containers which are not in the 'kube-system' namespace are checkpointed.

krmayankk · 2021-01-05T09:03:30Z

staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto

+    // Keep temporary files. Like log files. Helpful for debugging.
+    bool keep = 1;
+    // Checkpoint/Restore the container with established TCP connections.
+    bool tcp_established = 2;


is it possible to checkpoint and restore with established tcp connections ? How does that work ?

is it possible to checkpoint and restore with established tcp connections ?

Yes it is possible to checkpoint and restore established TCP connections. On the lower levels, checkpointing single processes or containers using runc/Podman it makes more sense when talking about the TCP connection from an outside client to the process or container. The restored process or container needs to have access to the same IP or the restore will fail.

In the Kubernetes context, especially for the current drain --checkpoint implementation, it might seem less useful for connections to external clients. It is, however, important to have the possibility to checkpoint and restore established TCP connections for containers and Pods which have open TCP connection between each other.

How does that work ?

Please have a look at how CRIU does that: https://criu.org/TCP_connection

riteshnaik · 2021-01-08T20:27:00Z

staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go

@@ -6964,6 +6964,395 @@ func (m *ReopenContainerLogResponse) XXX_DiscardUnknown() {

 var xxx_messageInfo_ReopenContainerLogResponse proto.InternalMessageInfo

+// Common options used for checkpointing and restoring.
+type CheckpointRestoreOptions struct {


How about adding an image directory option for a custom checkpoint storage directory that checkpoint and restore could use to save the checkpoint and restore from the checkpoint respectively?

See the archive option. A container checkpoint is much more than just the result of running CRIU. Then a directory would make sense. To make life easier for users I put everything in a tar archive in Podman. The CRIU checkpoint, the changes to the container file system and metadata (like name, IP address, ID and much more). Without those additional information it is difficult to restore a container as it was before without requiring a lot of manual steps from the user during restore. The archive contains everything. Please also see me CRI-O implementation where I either put all container information as described above in a tar archive or even a complete Pod checkpoint containing all information about a Pod to be restored.

Ah...I missed the archive option here. Thanks for clarifying and explaining the details behind the archive option. I started by looking to check if there is an option to restore multiple containers from the same checkpoint and seems like there is. Restore has to be just pointed to the archive using the import option and given a new name via the name option.
I did go through the Podman implementation and as I mentioned earlier it did answer a lot of my questions. I haven't had the chance to go over the CRI-O implementation yet and will definitely go over it to learn more on a complete Pod checkpoint workflow.

riteshnaik · 2021-01-08T20:28:14Z

staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go

@@ -6964,6 +6964,395 @@ func (m *ReopenContainerLogResponse) XXX_DiscardUnknown() {

 var xxx_messageInfo_ReopenContainerLogResponse proto.InternalMessageInfo

+// Common options used for checkpointing and restoring.


Wondering what's the plan for adding more options in the future? Is this the exhaustive list or plan is to start with some basic options and to add more based on the use cases? Are these options based on options that are available for criu?

The list of options is based on my work on Podman. All the features I implemented there I added here as they seem to be the most interesting to begin with. It also depends on what runc/crun can do. One of the things runc can currently do which is not represented here is pre-copy and post-copy migration. Which is nice to have in the future but I do not think it is important to think about it now.

Going through your work on adding checkpoint/restore support for Podman helped me put things in perspective and answered my questions. Thanks!

ehashman

/hold
since this is a POC PR for a KEP which hasn't been approved

/ok-to-test

k8s-ci-robot · 2021-02-25T09:06:12Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: adrianreber
To complete the pull request process, please assign derekwaynecarr, liggitt after the PR has been reviewed.
You can assign the PR to them by writing /assign @derekwaynecarr @liggitt in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

hack/OWNERS
staging/src/k8s.io/cri-api/pkg/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

adrianreber · 2021-02-25T09:14:00Z

/test pull-kubernetes-node-e2e

adrianreber · 2021-02-25T09:40:22Z

/test pull-kubernetes-node-e2e

adrianreber · 2021-02-25T09:41:14Z

/test pull-kubernetes-bazel-test

adrianreber · 2021-02-25T09:47:56Z

/test pull-kubernetes-node-e2e

First user of this experimental CRI API is checkpoint/restore. The idea behind this experimental API is that CRI implementations do not have to implement this but it makes it possible to easily test new features which require additions to the CRI API. Signed-off-by: Adrian Reber <areber@redhat.com>

Signed-off-by: Adrian Reber <areber@redhat.com>

adrianreber · 2021-02-25T10:38:41Z

/test pull-kubernetes-node-e2e
/test pull-kubernetes-e2e-kind

k8s-ci-robot · 2021-02-25T10:42:47Z

@adrianreber: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-azure-file	4b922f8950c497663e3abfa7f943ca2160373a10	link	`/test pull-kubernetes-e2e-azure-file`
pull-kubernetes-e2e-azure-disk	4b922f8950c497663e3abfa7f943ca2160373a10	link	`/test pull-kubernetes-e2e-azure-disk`
pull-kubernetes-e2e-azure-disk-windows	4b922f8950c497663e3abfa7f943ca2160373a10	link	`/test pull-kubernetes-e2e-azure-disk-windows`
pull-kubernetes-e2e-azure-file-windows	4b922f8950c497663e3abfa7f943ca2160373a10	link	`/test pull-kubernetes-e2e-azure-file-windows`
pull-kubernetes-e2e-aks-engine-azure	4b922f8950c497663e3abfa7f943ca2160373a10	link	`/test pull-kubernetes-e2e-aks-engine-azure`
pull-kubernetes-e2e-azure-disk-vmss	4b922f8950c497663e3abfa7f943ca2160373a10	link	`/test pull-kubernetes-e2e-azure-disk-vmss`
pull-kubernetes-e2e-gce-alpha-features	4b922f8950c497663e3abfa7f943ca2160373a10	link	`/test pull-kubernetes-e2e-gce-alpha-features`
pull-kubernetes-e2e-gce-network-proxy-grpc	135d554265d151ea696992259f739721f0164758	link	`/test pull-kubernetes-e2e-gce-network-proxy-grpc`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

adrianreber · 2021-02-25T12:14:35Z

/test pull-kubernetes-node-e2e

k8s-ci-robot · 2021-04-06T02:12:49Z

@adrianreber: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

enj · 2021-04-19T14:14:18Z

/remove-sig auth

Feel free to add sig-auth back if this needs our attention.

fejta-bot · 2021-07-18T14:15:40Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

k8s-triage-robot · 2021-08-17T14:38:52Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2021-09-16T15:15:07Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2021-09-16T15:15:24Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 4, 2021

k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Jan 4, 2021

k8s-ci-robot requested review from dims and krmayankk January 4, 2021 17:03

adrianreber mentioned this pull request Jan 4, 2021

[WIP] Add --checkpoint to drain #97194

Closed

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 4, 2021

adrianreber mentioned this pull request Jan 4, 2021

Add Forensic Container Checkpointing KEP kubernetes/enhancements#1990

Merged

k8s-ci-robot assigned derekwaynecarr and mrunalp Jan 4, 2021

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 4, 2021

krmayankk reviewed Jan 5, 2021

View reviewed changes

ehashman added this to Needs Reviewer in SIG Node PR Triage Jan 6, 2021

ehashman moved this from Needs Reviewer to Triage in SIG Node PR Triage Jan 6, 2021

riteshnaik reviewed Jan 8, 2021

View reviewed changes

tfenster mentioned this pull request Jan 25, 2021

Question: Support for docker checkpoint on Windows? microsoft/Windows-Containers#88

Closed

ehashman reviewed Jan 26, 2021

View reviewed changes

adrianreber changed the title ~~Extend CRI API for checkpoint/restore~~ Add experimental CRI API for checkpoint/restore Feb 12, 2021

tfenster mentioned this pull request Feb 17, 2021

Support for checkpoint / restore on Windows as well #99170

Closed

adrianreber force-pushed the 2021-01-04-cri-api branch from f72e9f0 to fe53b12 Compare February 25, 2021 09:05

adrianreber added 2 commits February 25, 2021 09:56

Experimental CRI API make update

a740072

Signed-off-by: Adrian Reber <areber@redhat.com>

adrianreber force-pushed the 2021-01-04-cri-api branch from fe53b12 to a740072 Compare February 25, 2021 09:57

adrianreber mentioned this pull request Mar 25, 2021

Pod man error2 checkpoint-restore/criu#1420

Closed

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 6, 2021

ritazh added this to In Progress in SIG Auth Old Apr 9, 2021

k8s-ci-robot removed the sig/auth Categorizes an issue or PR as relevant to SIG Auth. label Apr 19, 2021

enj removed this from Needs Triage PRs in SIG Auth Old Apr 19, 2021

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 18, 2021

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Aug 17, 2021

k8s-ci-robot closed this Sep 16, 2021

SIG Node PR Triage automation moved this from Waiting on Author to Done Sep 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add experimental CRI API for checkpoint/restore #97689

Add experimental CRI API for checkpoint/restore #97689

adrianreber commented Jan 4, 2021 •

edited

k8s-ci-robot commented Jan 4, 2021

k8s-ci-robot commented Jan 4, 2021

fejta-bot commented Jan 4, 2021

dims commented Jan 4, 2021

fejta-bot commented Jan 4, 2021

krmayankk Jan 5, 2021

adrianreber Jan 5, 2021

krmayankk Jan 12, 2021

adrianreber Jan 12, 2021

krmayankk Jan 5, 2021

adrianreber Jan 5, 2021

riteshnaik Jan 8, 2021

adrianreber Jan 9, 2021

riteshnaik Jan 11, 2021

riteshnaik Jan 8, 2021

adrianreber Jan 9, 2021

riteshnaik Jan 11, 2021

ehashman left a comment

k8s-ci-robot commented Feb 25, 2021

adrianreber commented Feb 25, 2021

adrianreber commented Feb 25, 2021

adrianreber commented Feb 25, 2021

adrianreber commented Feb 25, 2021

adrianreber commented Feb 25, 2021

k8s-ci-robot commented Feb 25, 2021 •

edited

adrianreber commented Feb 25, 2021

k8s-ci-robot commented Apr 6, 2021

enj commented Apr 19, 2021

fejta-bot commented Jul 18, 2021

k8s-triage-robot commented Aug 17, 2021

k8s-triage-robot commented Sep 16, 2021

k8s-ci-robot commented Sep 16, 2021

		@@ -6964,6 +6964,395 @@ func (m *ReopenContainerLogResponse) XXX_DiscardUnknown() {

		var xxx_messageInfo_ReopenContainerLogResponse proto.InternalMessageInfo

		// Common options used for checkpointing and restoring.

Add experimental CRI API for checkpoint/restore #97689

Add experimental CRI API for checkpoint/restore #97689

Conversation

adrianreber commented Jan 4, 2021 • edited

k8s-ci-robot commented Jan 4, 2021

k8s-ci-robot commented Jan 4, 2021

fejta-bot commented Jan 4, 2021

dims commented Jan 4, 2021

fejta-bot commented Jan 4, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ehashman left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 25, 2021

adrianreber commented Feb 25, 2021

adrianreber commented Feb 25, 2021

adrianreber commented Feb 25, 2021

adrianreber commented Feb 25, 2021

adrianreber commented Feb 25, 2021

k8s-ci-robot commented Feb 25, 2021 • edited

adrianreber commented Feb 25, 2021

k8s-ci-robot commented Apr 6, 2021

enj commented Apr 19, 2021

fejta-bot commented Jul 18, 2021

k8s-triage-robot commented Aug 17, 2021

k8s-triage-robot commented Sep 16, 2021

k8s-ci-robot commented Sep 16, 2021

adrianreber commented Jan 4, 2021 •

edited

k8s-ci-robot commented Feb 25, 2021 •

edited