Init KEP to migrate in-tree dockershim to out-of-tree #866

resouer · 2019-03-01T22:48:26Z

A initial KEP to discuss about criteria and plan to migrate in-tree dockershim to out-of-tree.

Please note this KEP is not target on 1.14 or any specific release recently, it's designed to be a join effort spanning on many future releases and requires carefully revise and thinking.

/assign @yujuhong @dchen1107

/cc @dims @kubernetes/sig-node-pr-reviews

dims · 2019-03-06T00:43:40Z

keps/sig-node/20190226-migrate-in-tree-dockershim-to-out-of-tree.md

+
+## Summary
+
+CRI for docker (i.e. dockershim) is currently a build-in container runtime in kubelet code base. This proposal aims at a concrete deprecation and migration plan for separating dockershim from kubelet to out-of-tree without breaking current production users and WIP engineering efforts.


build-in -> built-in

dims · 2019-03-06T00:47:23Z

keps/sig-node/20190226-migrate-in-tree-dockershim-to-out-of-tree.md

+- Updating all the eco-system tools to support the new cri-dockerd.
+- Many people use the built in dockershim for in-cluster image build. While that may not be something we recommend for a variety of reasons, it will be a breaking change for these users.
+- CRI is still in alpha，should probably get a 1.0 out there splitting out dockershim completely from kubelet.
+- Existed CNI and CSI plugins may also be affected.


Existing instead of Existed

dims · 2019-03-06T00:49:34Z

keps/sig-node/20190226-migrate-in-tree-dockershim-to-out-of-tree.md

+- Many people use the built in dockershim for in-cluster image build. While that may not be something we recommend for a variety of reasons, it will be a breaking change for these users.
+- CRI is still in alpha，should probably get a 1.0 out there splitting out dockershim completely from kubelet.
+- Existed CNI and CSI plugins may also be affected.
+


cri-dockerd will still be vendoring kubernetes/kubernetes for a while during the transition. vendoring kubernetes/kubernetes is not easy

ACK, it's a good point

dims · 2019-03-06T00:51:58Z

keps/sig-node/20190226-migrate-in-tree-dockershim-to-out-of-tree.md

+- A concrete dockershim deprecation criteria.
+- A brief plan to deprecate dockershim spanning multiple releases.
+- Issues need to be fixed in kubelet in order to separate dockershim from it.
+


Can we add another goal of writing up a criteria for adopting one of the existing CRI runtimes (for example containerd) that can eventually replace cri-dockerd?

considering that there are too many docker users, it be better not to cease cri-dockerd here.

SIG Node doesn't want to make a call which CRI implementation the vendor / the user should use. Instead, we design the API to define how the runtime engaging with Kubelet and nodes, a test suite to verify / qualify the functionalities, a portable toolset (cri-tool) for debuggability and introspect.

dims · 2019-03-06T00:53:25Z

keps/sig-node/20190226-migrate-in-tree-dockershim-to-out-of-tree.md

+- Mark in-tree dockershim as "maintenance mode":
+  - CRI generic changes/features can continue on dockershim.
+  - WIP efforts on dockershim can continue and go to complete.
+  - dockershim/docker specific changes/features should be rejected.


we will continue updating the docker/docker dependency. right?

Perhaps continue updating the docker/docker dependency is a must.

Yes, we still need to update Docker when needed.

dims · 2019-03-22T03:17:21Z

@yujuhong @derekwaynecarr @PatrickLang @dchen1107 does this seem ready? (can we merge and iterate?)

dims · 2019-03-26T16:01:20Z

Another benefit as noted by @liggitt in the dependency doc[1], docker has just 7 month support[2]. So the quicker we get off dockerd shim the better we will be.

[1] https://docs.google.com/document/d/1WA8N7C48nkJmme9a96DU0o9jBpeycPhht8WF-Eam9QQ/edit?ts=5c9a308b#heading=h.46nlk76xzfpg
[2] https://docs.docker.com/install/#support

dims · 2019-03-28T12:58:27Z

keps/sig-node/20190226-migrate-in-tree-dockershim-to-out-of-tree.md

+  - support to get container logs when docker uses journald as the driver.
+  - logic of moving docker processes to a given cgroup
+  - TBD anything else?
+- Package in-tree dockershim separately from kubelet and provide a "option" to enable/disable it.


@resouer let's please add a "new binary named cri-dockerd" with code being in-tree (line number 113)

We will also need to ensure that this binary gets into the release artifacts

@dims do you mean that cri-dockerd could be compiled to an independent binary like kubelet here?

@zhangxiaoyu-zidif yep.

liggitt · 2019-03-28T13:13:30Z

Long-term, do we want to maintain a dockershim CRI implementation? If not, would it make more sense to focus efforts on getting CRI to v1, then immediately start the deprecation period for dockershim (6 months, according to https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-a-flag-or-cli) in favor of existing out of tree CRI impls?

liggitt · 2019-03-28T13:09:36Z

keps/sig-node/20190226-migrate-in-tree-dockershim-to-out-of-tree.md

+- Package in-tree dockershim separately from kubelet and provide a "option" to enable/disable it.
+- Ensure e2e/Node e2e test framework is CRI generic and test cases are independent of container runtime.
+
+Step 2: Work out a out-of-tree CRI for docker


does this not already exist in the form of containerd?

I assume they are different. During recent sig-node meeting, folks from Docker Inc are willing to maintain a CRI-docker.

All recent versions of docker are built on top of containerd, so I'm unclear on what functionality dockershim has that cri-containerd does not.

@Random-Liu do you know if there is any reason to use dockershim over cri-containerd with recent versions of docker?

I think in the long run, we do expect people to move to CRI conformant container runtimes like containerd and cri-o to be able to leverage new Kubernetes features.

However, people may still need more time to make that move, one of the biggest reason is that with dockershim, you can actually see all your Kubernetes containers in Docker, and people's tools may have been relying on that. For example, people may have built security tools, monitoring tools, logging tools around that.

With containerd or cri-o, you won't see Kubernetes containers in Docker any more. It takes time for people to migrate their tooling, that's why we need to keep dockershim for a while.

Another reason is that today's Kubernetes windows support is still based on dockershim. cri-containerd Windows is WIP, but not finished yet.

so we cloud remain both mechanisms currently and depreciated the old way gradually.

Yes, there are still features that are supported by docker/dockershim, but not with containerd.

liggitt · 2019-03-28T13:12:09Z

keps/sig-node/20190226-migrate-in-tree-dockershim-to-out-of-tree.md

+- The number of affected users maybe large.
+- Updating all the eco-system tools to support the new cri-dockerd.
+- Many people use the built in dockershim for in-cluster image build. While that may not be something we recommend for a variety of reasons, it will be a breaking change for these users.
+- CRI is still in alpha，should probably get a 1.0 out there splitting out dockershim completely from kubelet.


I strongly support progressing CRI to v1, for the sake of all the runtime implementations. I'm less clear on the benefit of extracting dockershim to a standalone CRI. Did you consider deprecating dockershim in favor of existing CRI implementations instead?

I will try to confirm if sig-node or Docker Inc has a plan to support a CRI-docker for long time (which is what I heard during the sig meeting). If not, deprecating dockershim directly will be the goal.

We talked about this at SIG Node, including the folks from Docker Inc. Docker folks plan to maintain cri-dockershim.

liggitt · 2019-03-28T13:14:22Z

keps/sig-node/20190226-migrate-in-tree-dockershim-to-out-of-tree.md

+
+Actions:
+
+- Refactoring e2e/Node e2e test framework to include CRI for docker installation (or use other CRI container runtime).


this would be extremely helpful for existing CRI impls to test/verify compatibility

Cluster/node e2e doesn't care what CRI implementation you use under the hood. I think maybe I've missed the point of this statement.

There're two points in my mind:

Ensure cluster/node e2e are 100% CRI focused. For instance: DockerValidator should use the CRI, not the Docker API kubernetes#55414

Ensure test-infra install CRI-docker or cri-containerd binary in e2e machines. Currently, they install Docker binary only.

dims · 2019-03-28T13:17:32Z

@resouer WDYT of @liggitt 's alternative? I am inclined to support that as it is less work overall

resouer · 2019-03-28T21:55:24Z

@dims I agree, will update the KEP to reflect this proposal.

tedyu · 2019-11-01T18:10:04Z

keps/sig-node/20190226-migrate-in-tree-dockershim-to-out-of-tree.md

+- kubelet has no dependency on dockershim/docker in its whole lifecycle. 
+- All node related features are CRI generic and have no "back door" dependency on dockershim/docker.
+- Deprecate and remove, or replace all Docker-specific features.
+- Reasonable benchmark result of performance degradation after moving dockershim to out-of-tree.


What percentage of degradation is acceptable (10%) ?

10% might not be acceptable by the wider audience.
but my guess is that it's going to end up being less than that 🤔

tedyu · 2019-11-01T20:22:01Z

Currently starting dockershim is governed by a flag:

			if kubeletServer.KubeletFlags.ExperimentalDockershim {
				if err := RunDockershim(&kubeletServer.KubeletFlags, kubeletConfig, stopCh); err != nil {

Would there be another flag or enum which is used to choose the effective out-of-tree container runtime ?
My assumption is that the kubeletServer.KubeletFlags, kubeletConfig and stopCh parameters provide sufficient information for starting the chosen out-of-tree container runtime.

bg-chun · 2020-01-05T21:49:06Z

/cc @bg-chun

bg-chun · 2020-01-05T21:55:47Z

How is it going?
For my work(container isolation of hugepages).
I updated CRI to have hugepages limit for containers then I am working for Container Runtimes.
Containerd and cri-o doesn't has any issue for my work But, Docker has tiny issue due to the dockershim dependency :( .
I jsut hope I will see this changes in 2020.

liggitt · 2020-01-22T16:13:50Z

cc @dashpole - what does kubelet/cadvisor do differently when configured to speak to a CRI instead of dockershim? does cadvisor still contact the container runtime directly or would this let us stop enabling runtime-specific integrations in cadvisor?

dashpole · 2020-01-22T17:42:03Z

what does kubelet/cadvisor do differently when configured to speak to a CRI instead of dockershim? does cadvisor still contact the container runtime directly or would this let us stop enabling runtime-specific integrations in cadvisor?

The only difference is that it disables metrics provided by the CRI. It still contacts the container runtime directly to get metadata about containers.

fejta-bot · 2020-04-21T17:52:27Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

warmchang · 2020-04-22T00:33:06Z

/remove-lifecycle stale

fejta-bot · 2020-07-21T01:05:38Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

frittentheke · 2020-07-22T05:22:05Z

/remove-lifecycle stale

BenTheElder · 2020-07-22T15:13:02Z

If @resouer is working on this PR it won't go stale, but just removing the stale label is not going to get this merged. If they're not working on it someone should probably start a forked PR.

saschagrunert · 2020-09-09T13:18:53Z

@resouer there are ongoing discussions about deprecating dockershim in Kubernetes. Are you around to drive this KEP forward or would you prefer if someone else takes over? 😇

dims · 2020-09-09T13:23:03Z

@saschagrunert after talking to @liggitt and @derekwaynecarr this plan is kinda unworkable. Please see prototype in https://github.com/dims/cri-dockerd

Problem is that this external CRI impl will need to vendor in k8s.io/kubernetes which is verboten! ( https://github.com/dims/cri-dockerd/blob/main/go.mod#L21 )

saschagrunert · 2020-09-09T13:33:27Z

@dims I'm fine with that as well as the planned deprecation. I guess we can still recycle some parts like the hostport manager.

dims · 2020-09-09T13:43:27Z

@saschagrunert for sure! if you see containerd/cri repo, it no longer depends on k/k :) we should do the same for CRI-O

saschagrunert · 2020-09-09T13:45:27Z

@saschagrunert for sure! if you see containerd/cri repo, it no longer depends on k/k :) we should do the same for CRI-O

Yes, thanks for supporting us on that front! ❤️

fejta-bot · 2020-12-08T13:55:20Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

warmchang · 2020-12-08T14:24:13Z

keps/sig-node/20190226-migrate-in-tree-dockershim-to-out-of-tree.md

+
+Step 1: Stabilize in-tree dockershim and decouple dockershim from kubelet (but still in-tree).
+
+Target releases: 1.15, 1,16, 1.17


target release is v1.20 now. 😄

SergeyKanzhelev · 2020-12-29T22:12:26Z

I think this is outdated after #2221 and #1985

/close

k8s-ci-robot · 2020-12-29T22:12:38Z

@SergeyKanzhelev: Closed this PR.

In response to this:

I think this is outdated after #2221 and #1985

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot assigned dchen1107 and yujuhong Mar 1, 2019

k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 1, 2019

k8s-ci-robot requested a review from dims March 1, 2019 22:48

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 1, 2019

resouer force-pushed the migrate-dockershim branch from 263793c to 05bf854 Compare March 5, 2019 22:05

dims reviewed Mar 6, 2019

View reviewed changes

This was referenced Mar 7, 2019

[WIP] Initial-cut of cri-dockerd kubernetes/kubernetes#74051

Closed

Extracting CRI API to a new staging repository kubernetes/kubernetes#75293

Closed

dims mentioned this pull request Mar 21, 2019

New kubernetes/cri-api github repository for staging the CRI API kubernetes/org#642

Closed

dims mentioned this pull request Mar 26, 2019

New staging repository for cri-api kubernetes/kubernetes#75531

Merged

dims mentioned this pull request Mar 27, 2019

Update docker/docker dependency in Kubernetes kubernetes/kubernetes#75774

Closed

dims reviewed Mar 28, 2019

View reviewed changes

liggitt reviewed Mar 28, 2019

View reviewed changes

aojea mentioned this pull request Apr 22, 2019

Add revised IPv4/IPv6 dual stack KEP #808

Merged

resouer force-pushed the migrate-dockershim branch from c2345c7 to ede8a32 Compare April 22, 2019 18:26

k8s-ci-robot added the sig/cli Categorizes an issue or PR as relevant to SIG CLI. label Apr 22, 2019

resouer force-pushed the migrate-dockershim branch from ede8a32 to 55d9c7c Compare April 22, 2019 18:27

tedyu reviewed Nov 1, 2019

View reviewed changes

k8s-ci-robot requested a review from bg-chun January 5, 2020 21:49

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 21, 2020

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 22, 2020

jayunit100 mentioned this pull request Jul 16, 2020

Fix bandwidth burst too large kubernetes/kubernetes#90839

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 21, 2020

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 22, 2020

jayunit100 mentioned this pull request Sep 6, 2020

Move hostport manager to staging or out of tree kubernetes/kubernetes#94538

Closed

dims mentioned this pull request Sep 10, 2020

Deprecate Dockershim kubernetes/kubernetes#94624

Merged

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 8, 2020

warmchang reviewed Dec 8, 2020

View reviewed changes

k8s-ci-robot closed this Dec 29, 2020


		## Summary

		CRI for docker (i.e. dockershim) is currently a build-in container runtime in kubelet code base. This proposal aims at a concrete deprecation and migration plan for separating dockershim from kubelet to out-of-tree without breaking current production users and WIP engineering efforts.


		Actions:

		- Refactoring e2e/Node e2e test framework to include CRI for docker installation (or use other CRI container runtime).


		Step 1: Stabilize in-tree dockershim and decouple dockershim from kubelet (but still in-tree).

		Target releases: 1.15, 1,16, 1.17

Init KEP to migrate in-tree dockershim to out-of-tree #866

Init KEP to migrate in-tree dockershim to out-of-tree #866

Conversation

resouer commented Mar 1, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dims Mar 6, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dims commented Mar 22, 2019

dims commented Mar 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt commented Mar 28, 2019 • edited Loading

Choose a reason for hiding this comment

resouer Apr 1, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Random-Liu Apr 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dims commented Mar 28, 2019

resouer commented Mar 28, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tedyu commented Nov 1, 2019 • edited Loading

bg-chun commented Jan 5, 2020

bg-chun commented Jan 5, 2020

liggitt commented Jan 22, 2020

dashpole commented Jan 22, 2020

fejta-bot commented Apr 21, 2020

warmchang commented Apr 22, 2020

fejta-bot commented Jul 21, 2020

frittentheke commented Jul 22, 2020

BenTheElder commented Jul 22, 2020

saschagrunert commented Sep 9, 2020

dims commented Sep 9, 2020

saschagrunert commented Sep 9, 2020

dims commented Sep 9, 2020

saschagrunert commented Sep 9, 2020

fejta-bot commented Dec 8, 2020

Choose a reason for hiding this comment

SergeyKanzhelev commented Dec 29, 2020

k8s-ci-robot commented Dec 29, 2020

dims Mar 6, 2019 •

edited

Loading

liggitt commented Mar 28, 2019 •

edited

Loading

resouer Apr 1, 2019 •

edited

Loading

Random-Liu Apr 20, 2019 •

edited

Loading

tedyu commented Nov 1, 2019 •

edited

Loading