Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate Dockershim #94624

Merged
merged 1 commit into from
Nov 12, 2020
Merged

Conversation

dims
Copy link
Member

@dims dims commented Sep 8, 2020

See KEP in progress - kubernetes/enhancements#1985

dockershim has been part of kubelet for far too long. we vendor in a whole bunch of stuff from moby/moby to make things work. The correct split/architecture is to follow the same design as with containerd and cri-o namely a separate process that kubelet talks to via grpc that in turn talks to a runtime. If folks are interested to do this, please do. An example of such a split is https://github.com/dims/cri-dockerd (NO! i am not maintaining that, nice try).

In this PR, we start the clock on removing dockershim in a subsequent release. We already have dockerless tag and CI jobs to ensure that we don't break anything when we eventually drop the built-in dockershim.

Signed-off-by: Davanum Srinivas davanum@gmail.com

What type of PR is this?
/kind cleanup
/kind deprecation

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Action Required: Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community. We encourage you to evaluate moving to a container runtime that is a full-fledged implementation of CRI (v1alpha1 or v1 compliant) as they become available.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Sep 8, 2020
@dims
Copy link
Member Author

dims commented Sep 8, 2020

/assign @dchen1107 @derekwaynecarr @dashpole

@k8s-ci-robot k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 8, 2020
@MHBauer
Copy link
Contributor

MHBauer commented Sep 8, 2020

As far as I can tell the only CI job is for typecheck, it's not tested with a built configuration, but I can't see how building without the code would break anything.
/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 8, 2020
@marosset
Copy link
Contributor

marosset commented Sep 8, 2020

The first containerd release with Windows support was 1.4.0 and just released in Aug 2020.
The Windows community has been actively investing in containerd support since 1.18 and will continue to do so but this is still new.

I'm concerned that marking dockershim as deprecated before ContainerD on Windows has had time to get any real adoption might cause some users to question Windows support in Kubernetes.

I'm in favor of deprecating dockershim and also removing that code from a Windows standpoint, I just think it might be better to do this in 1.21.

/cc @michmike

@adammw
Copy link

adammw commented Sep 8, 2020

We now have robust implementations of the CRI interface

Is there a CRI-based implementation for docker? I haven't been able to find any guides on how to migrate to that - could someone link me to docs around that?

@michmike
Copy link
Contributor

michmike commented Sep 8, 2020

The first containerd release with Windows support was 1.4.0 and just released in Aug 2020.
The Windows community has been actively investing in containerd support since 1.18 and will continue to do so but this is still new.

I'm concerned that marking dockershim as deprecated before ContainerD on Windows has had time to get any real adoption might cause some users to question Windows support in Kubernetes.

I'm in favor of deprecating dockershim and also removing that code from a Windows standpoint, I just think it might be better to do this in 1.21.

/cc @michmike

100% agree with Mark. we need to wait for containerd support in windows to stabilize and iron out any kinks. 1.21 is the right timeframe

@neolit123
Copy link
Member

neolit123 commented Sep 8, 2020

The kubelet has a built-in dockershim which was used to support Docker. We now have robust implementations of the CRI interface, so we will be removing dockershim in a subsequent release

possibly requires some clarification - are we removing it entirely or just from the kubelet?

moving dockershim outside of the kubelet to a community maintained repository seems the right action here. survey from KubeCon EU (this year) performed by SIG Cluster Lifecycle:
https://docs.google.com/forms/d/1AA75xUUdsboenrGPFN-Dtnx8N8vfR_b3kmmQNrdXX4g/edit#responses

search for "What is your preferred container runtime (CRI)?". Docker still has majority but containerd is gaining traction slowly.

@dims
Copy link
Member Author

dims commented Sep 9, 2020

@marosset @michmike Starting the countdown does not imply we have to remove things in a specific time period. The more heads up people get the better. So i'd rather throw this out earlier (in 1.20) than later (1.21).

@dims
Copy link
Member Author

dims commented Sep 9, 2020

@MHBauer we had added a kind CI job that used the dockerless tag. Did that get removed?

@liggitt
Copy link
Member

liggitt commented Sep 9, 2020

@marosset @michmike Starting the countdown does not imply we have to remove things in a specific time period. The more heads up people get the better. So i'd rather throw this out earlier (in 1.20) than later (1.21).

I agree

@neolit123
Copy link
Member

neolit123 commented Sep 9, 2020

as mentioned, i don't entirely agree with the plan here...this PR effectively deprecates Docker support in Kubernetes, but the release note does not communicate this clearly.

what would be guide to explain to the users how to move their cluster from docker to e.g. containerd?
immutable node upgrades are maybe fine, but what about in-place on limited # bare metal machines? matching cgroup drivers? state cleanup?

my proposal:

Action Required: Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community. We encourage you to move to a container runtime that is a full-fledged implementation of CRI instead. To transition your existing clusters follow this guide

and the "this guide" document has to be created, before we apply this deprecation.

pkg/kubelet/kubelet.go Outdated Show resolved Hide resolved
@dims
Copy link
Member Author

dims commented Sep 9, 2020

@neolit123 i'd prefer the community to write the "guide". I am just interested in starting the clock in this PR. I updated the release note with the rest of your text.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

1 similar comment
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@dhiltgen
Copy link

dhiltgen commented Dec 2, 2020

A number of folks have commented about still relying on docker build in their environments. There are various projects and approaches to build on a kubernetes cluster, but if you happen to be looking for a "drop in" replacement for the docker build UX, you might consider https://github.com/vmware-tanzu/buildkit-cli-for-kubectl/ as a possible option to consider (which works great on k8s+containerd)

@IAXES
Copy link

IAXES commented Dec 2, 2020

Good day,

In simple/layperson terms, does this mean that Docker containers will not be supported at all in the future? For example, Docker's landing page notes that it uses containerd (i.e. "Containerd is used by Docker, Kubernetes CRI, and a few other projects" [1]). Is containerd able to effectively work as a drop-in replacement? If not, my understanding is that, if we want to use newer builds of K8S going forward (i.e. once the deprecated features are removed outright), individuals/teams would have to (please feel free to correct any of these points if they're off the mark):

  • Change how they build images, i.e.:
    • Re-write build scripts to be something other than Dockerfile.
    • Use tooling other than docker build.
    • Use different image registry tools for publishing images.

If the above assumptions are correct, are there de-facto (i.e. most commonly used in the K8S community, or recommended by K8S developers and maintainers) replacements for Docker (i.e. in terms of tooling, publishing, etc.) so individuals/teams can migrate effectively?

Thank you.

  1. "What is containerd?", https://www.docker.com/blog/what-is-containerd-runtime/, last accessed 2020-12-02.

@BenTheElder
Copy link
Member

BenTheElder commented Dec 2, 2020

In simple terms (I hope, can further clarify if needed): docker containers / images will absolutely be supported. docker may not be the tool Kubernetes uses to run them, but the same Kubernetes YAML, docker images, etc. will continue to work with other container runtimes.

Images use an open format common to all runtimes, you don't need to change image builds.

It should be a drop in replacement. Our (Kubernetes) CI uses them interchangably. If you've ever used microk8s, k3s, or KIND then you've already used containerd to run Kubernetes workloads.

This should only affect cluster operators / implementors.

@IAXES
Copy link

IAXES commented Dec 2, 2020

In simple terms (I hope, can further clarify if needed): docker containers / images will absolutely be supported. docker may not be the tool running them, but the same Kubernetes YAML, docker images, etc. will continue to work with other container runtimes.

Images use an open format common to all runtimes, you don't need to change image builds.

It should be a drop in replacement. Our (Kubernetes) CI uses them interchangably. If you've ever used microk8s, k3s, or KIND then you've already used containerd to run Kubernetes workloads.

This should only affect cluster operators / implementors,

Thank you very much. This is indeed helpful.

@neolit123
Copy link
Member

neolit123 commented Dec 3, 2020

@dims i'd add a message in the release note that this is not deprecating support for Docker containers...(given the high number of questions about this).
the next (pre-)release will pick the updated release note.

proposing the following update:

Action Required: Docker support as a container runtime in the kubelet is now deprecated and will be removed in a future release. Your Docker containers will continue to work. The kubelet uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community. We encourage you to evaluate moving to a container runtime that is a full-fledged implementation of CRI (v1alpha1 or v1 compliant) as they become available.

EDIT: might be a good idea to link to the KEP under this too:

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1985-remove-dockershim

EDIT/META: possibly link to the new blog post too.
https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/

@adamparco
Copy link

A happy update: Mirantis and Docker have partnered on supporting the dockershim as a standalone component outside of Kube. You can read more about it here: https://www.mirantis.com/blog/mirantis-to-take-over-support-of-kubernetes-dockershim-2/. Hopefully this will put any remaining lingering concerns to rest.

@dims
Copy link
Member Author

dims commented Dec 6, 2020

@adamparco thanks for the good news, let's chat about a migration plan and testing strategies to make sure we are able to support folks who will want to stick to using docker in the future.

For those who will check on this KEP, the plan is still in progress :) i.e, investigate and switch over to a CRI implementation that's available in the community (containerd, cri-o etc).

If you want to continue to use docker, You will need to switch to new stuff that will show up in https://github.com/Mirantis/cri-dockerd as we will drop dockershim per the timetable in the KEP.

Thanks!

@zhucan
Copy link
Member

zhucan commented Dec 9, 2020

@dims Is it needed to install docker-ce when install kubernetes, if docker-shim been deprecated .

@dims
Copy link
Member Author

dims commented Dec 9, 2020

@xlgao-zju
Copy link
Contributor

/cc

@k8s-ci-robot k8s-ci-robot requested a review from xlgao-zju March 26, 2021 07:06
@dims
Copy link
Member Author

dims commented Mar 26, 2021

Please see https://github.com/Mirantis/cri-dockerd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. sig/node Categorizes an issue or PR as relevant to SIG Node. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.