Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dockershim removal feedback & issues #106917

Closed
mrbobbytables opened this issue Dec 9, 2021 · 37 comments
Closed

Dockershim removal feedback & issues #106917

mrbobbytables opened this issue Dec 9, 2021 · 37 comments
Labels
area/docker lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. sig/docs Categorizes an issue or PR as relevant to SIG Docs. sig/node Categorizes an issue or PR as relevant to SIG Node. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@mrbobbytables
Copy link
Member

Please use this issue for feedback and issues regarding dockershim removal in the 1.24 release.

/triage accepted
/milestone v1.24
/sig docs node
/area docker

@afbjorklund

This comment was marked as outdated.

@afbjorklund
Copy link

afbjorklund commented Dec 10, 2021

Please use this issue for feedback and issues regarding dockershim removal in the 1.24 release.

This information is confusing: https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/

"This section presents information you need to know when migrating from dockershim to other container runtimes."

Shouldn't it be "when migrating from dockershim to other CRI implementations for Docker" ?

i.e. going from dockershim to cri-dockerd

"It is recommended to migrate from dockershim to alternative container runtimes."

Shouldn't it be "it is recommended to migrate from dockerd to alternative container runtimes" ?

i.e. going from dockerd to containerd


We saw the same thing happen with CRI-O, before. As the saying goes: "now you have two problems".

It is especially confusing when the docs still installs containerd from docker.com instead of containerd.io.

Moving from dockershim to cri-dockerd should be mandatory, since CRI will be required in 1.24 and up.

Moving from dockerd to containerd (or cri-o) could still be optional, and requires more user intervention.

@neolit123
Copy link
Member

neolit123 commented Dec 10, 2021

A transition to cri-dockerd first might be smoother, indeed. There is especially a concern for business with a lot of clusters, with a lot of docker nodes. If they cannot move to alternative runtimes due to bugs, these bugs will have to be reported and resolved. Requiring things like a host OS update with the CR update is not great.

cri-dockerd is currently a third party project with an undefined maintenance state. Because of that it is difficult to maintain a k8s.io guide for it. The new guide for migrating to containerd seems like the best option for the time being.

At the same time we don't want users to be stuck with a 1.23 kubelet. Note: the k8s skew policy allows nodes to continue to have a 1.23 kubelet against a 1.24 and 1.25 kube-apiserver, but that would only delay the required changes.

https://kubernetes.io/releases/version-skew-policy/#kubelet

@afbjorklund
Copy link

afbjorklund commented Dec 10, 2021

If I understand correctly, the plan is to remove all support (for docker) from the documentation and from the supporting tools like kubeadm and leave users to make their own adventure (or to pay for Docker or Mirantis)

  • No documentation on kubernetes.io
  • No testing in kubernetes regression
  • No binary packages available
  • No support in kubeadm init

For minikube it seems like continuing with docker runtime is doable, even if it means having to build our own CRI binaries and do our own CRI configuration. We do have containerd available as opt-in, since a long time back.

@neolit123
Copy link
Member

neolit123 commented Dec 10, 2021

If I understand correctly, the plan is to remove all support (for docker) from the documentation and from the supporting tools like kubeadm and leave users to make their own adventure (or to pay for Docker or Mirantis)

if cri-dockerd becomes a maintained and documented project, i don't see a problem for k8s.io to link and mention it as part of CR setup. https://kubernetes.io/docs/setup/production-environment/container-runtimes/

No documentation on kubernetes.io

cross-linking from k8s.io to cri-dockerd guides still make sense.

No testing in kubernetes regression

testing is difficult if the CRI implementation is external and unmaintained. if cri-dockerd ramp up on the maintenance i can see kubeadm adding back e2e tests for docker. we switched all test jobs to containerd for the time being.

No binary packages available

cri-dockerd need to start making releases and have a support matrix against the kubelet especially WRT the CRI API version that is in use by both.

No support in kubeadm init

kubeadm init|join allows you to pass any custom CRI socket via Init|JoinConfiguration.nodeRegistration.criSocket.
the CR will be treated as "remote" and configured (via a kubelet flags) similarly to how containerd | cri-o users, use kubeadm today. if docker.sock is present on the host and if cri-dockerd setups the legacy dockershim.sock, kubeadm would autodetect docker as the CR and start using it (we still have to make one minor change for kubeadm 1.24).

For minikube it seems like continuing with docker runtime is doable, even if it means having to build our own CRI binaries and do our own CRI configuration. We do have containerd available as opt-in, since a long time back

i would not recommend for minikube and users of k8s to start using cri-dockerd until the project is actually maintained, has docs and releases. if Mirantis cannot staff the project, they'd have to grant community ownership.

@afbjorklund
Copy link

afbjorklund commented Dec 10, 2021

I think Mirantis had plenty of time for that, with the delay... Don't think moving to 1.25 would have helped ?

Was referring to the auto-detection and the recommendations, it does work to use kubeadm with params

if cri-dockerd setups the legacy dockershim.sock

cri-dockerd now sets up /var/run/cri-docker.sock

EDIT: path changed to /var/run/cri-dockerd.sock


Users of minikube are free to choose their kubernetes version and the container runtime, using parameters..

If they want to stick with k8s 1.23, or move to containerd, they are free to do so. Otherwise, it is inevitable.
i.e. moving to version 1.24 and sticking with docker, without using cri-dockerd, is not going to be doable.

But it doesn't look to be too much worse than cri-o... Tested in another product only, scarce releases, etc.

@rphillips
Copy link
Member

Linking my issue about local-up-cluster.sh not working nicely due to the dockershim removal: #106972

@afbjorklund

This comment has been minimized.

@afbjorklund

This comment was marked as outdated.

@afbjorklund

This comment was marked as outdated.

@jyotimahapatra
Copy link
Contributor

@mrbobbytables 👋 I'm in the release 1.24 bug triage team. Since this issue is marked as a release blocker , want to find out if this is a placeholder issue? Let me know how we could track the progress as we go deeper into the release timeline.

@afbjorklund
Copy link

afbjorklund commented Jan 28, 2022

Currently there are no (working) upstream instructions for how to install the CRI for containerd and for docker.

There are inline instructions for containerd (downloading from docker.com, not from containerd.io) and for CRI-O.
But for cri-dockerd, there are no inline instructions and the upstream instructions have a fatal typo (and no packages)

It's not blocking, but it's not ideal. You would want the Kubernetes page to either have good info or link to good info ?


https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ -->

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-runtime -->

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

EDIT: I think it is a real risk, trying to use a corporate website for project documentation. Should have "cri-dockerd.io"
We already saw it happening with docs removed from docs.docker.com, it could happen also to docs.mirantis.com ?

@afbjorklund
Copy link

afbjorklund commented Feb 12, 2022

Adding a flag to minikube, to make it easier to test the new CRI with any version of Kubernetes (since 1.20 or so):

minikube start --kubernetes-version=v1.24.0-alpha.2 --container-runtime=docker \
               --cri-socket=/var/run/cri-dockerd.sock --cri-service=cri-docker.service

Currently we are bundling the pre-release cri-dockerd, until there can be an official v0.2.1 with the socket fixes.

cri-dockerd v0.2.0-11-ga4d1895
(Mirantis/cri-dockerd@a4d1895)


The default values for --cri-socket and --cri-service should otherwise work, matching the k8s version:

Kubernetes 1.23 and earlier: dockershim (internal)
Kubernetes 1.24 and later: cri-dockerd (external)

And yes, you can choose to run --container-runtime=containerd or --container-runtime=cri-o if you prefer.

They always run the external (remote) socket, since there was no internal "shim" for these two container runtimes.

@reylejano
Copy link
Member

Linking the Docs Dockershim Removal project board
https://github.com/orgs/kubernetes/projects/67

@afbjorklund
Copy link

Looks like "kubenet" is still available in cri-dockerd, maybe spoke too soon after reading those other issues...

Perhaps it wouldn't be a bad idea to start using CNI anyway, and use a "bridge" example like those other CR ?

@JamesLaverack
Copy link
Member

Hey all.

As we approach the release for Kubernetes 1.24 (scheduled just under a month away on 19th April 2022), we're keeping track of this issue in the release team. Currently this is tracked as a release blocker, but after discussions with the SIG Docs team (cc @nate-double-u) we're under the impression that the docs are in a good enough ™️ state to no longer be blocking.

I'd like to get community feedback on removing the release-blocker label from this issue, in particular:

  • Are the docs in a state in which we'd be happy to release with, even if not everything was complete for the day?
  • Is there anything else that we need to do for the Dockershim removal that would be a blocker for 1.24 if we didn't do it?

If people are happy, then I'll go ahead and remove the label.

@dims
Copy link
Member

dims commented Mar 23, 2022

@justaugustus you had marked this release-blocker way back in December. Looks like we are in much better shape now. So we should remove the release-blocker tag. please revisit this when you get a chance (if you still think this is one).

@JamesLaverack +1 to remove release-blocker

@mrbobbytables
Copy link
Member Author

I would also +1 removing release-blocker. The dockershim docs team has been doing a great job at shoring up the needed docs :)

@neolit123
Copy link
Member

neolit123 commented Mar 23, 2022

the decision seems up to release, but there are a number of open PRs (maybe some are not tracked in https://github.com/orgs/kubernetes/projects/67):

kubernetes/website#30882
kubernetes/website#32131

first one is rather important and has been in review since December.
without these merging prior to release there is potential for a flow of tickets to be logged in k/website and k/k.
as someone who does triage every week, i would like to avoid that...

@afbjorklund
Copy link

afbjorklund commented Mar 23, 2022

I will probably revert my change to stop using "kubenet", since it is still available (unlike indications to the contrary).
There are a lot of other flags removed from the kubelet that affects it, so it will need some code changes anyway*...

* those other code changes are not until 1.27 though

Hopefully we will have time to fix minikube before the beta release, so that it works (i.e. with docker) without them ?
The old code was relying on --network-plugin and --cni-conf-dir, both available as alpha features in cri-dockerd.

I think we can live with the other issues such as no release and no documentation, at least for 1.24 (maybe not 1.25)

But it seems like all warnings about trying to use it were warranted, and we better hurry up with the nerdctl tooling.

cri-dockerd is currently a third party project with an undefined maintenance state.

"Docker cri networking managed by network plugin kubernetes.io/no-op".


Plan B is to stick with the current k8s releases (1.23 and 1.24) and utilizing the version skew policy to get past 1.25.

So if you still require "docker" container runtime, you will get an older version of k8s when compared to "containerd".

@vsxen
Copy link

vsxen commented Mar 25, 2022

In addition, remove kubeadm check for socat & netfilter ?

	[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
	[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

@neolit123
Copy link
Member

@vsxen are you sure these are dockershim only?
See the containerd docs https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd

Also socat and iptables forwarding are different things, AFAIK. Happy to cleanup some if these checks but we need proof.

@vsxen
Copy link

vsxen commented Mar 25, 2022

I only find socat in there

https://github.com/kubernetes/kubernetes/blob/v1.23.0/pkg/kubelet/dockershim/docker_streaming_others.go#L44

for br_ netfilter it depends on which cni plugin you use, for example some cni plugin (cilium ovs) don't use bridge.

@dims
Copy link
Member

dims commented Mar 26, 2022

removing release-blocker

@neolit123
Copy link
Member

neolit123 commented Mar 28, 2022

@vsxen

I only find socat in there

We could move all dockershim dependencies to preflight warnings instead of errors in 1.24. We can also remove them, but cri-dockerd must document its dependencies first.

for br_ netfilter it depends on which cni plugin you use, for example some cni plugin (cilium ovs) don't use bridge.

This one is trickier. If some CNI (which ones? are they 'popular'?) need it, it might be better to continue throwing errors. Preflight errors can be ignored with the CLI flag / config.

@afbjorklund
Copy link

afbjorklund commented Mar 28, 2022

It seems the developer documentation for cri-dockerd gives just enough information to get started, but not to complete...

https://github.com/Mirantis/cri-dockerd#build-and-install

Missing some requirements like os version and go version, and missing the version linker flags that are provided by make

@ehashman
Copy link
Member

/milestone clear
as this is not actionable for the 1.24 release

/lifecycle frozen
as we will want to continue collecting feedback for some time :)

@k8s-ci-robot k8s-ci-robot added the lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. label Mar 30, 2022
@k8s-ci-robot k8s-ci-robot removed this from the v1.24 milestone Mar 30, 2022
@chrisnegus chrisnegus moved this from Unassigned To Do (Needs People) to TBD - Post-1.24 in Docs - Dockershim Removal Apr 12, 2022
@afbjorklund
Copy link

The docker runtime has now changed the default to "kubenet", which means that user needs to install CNI plugins.

NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: could not locate kubenet required CNI plugins [bridge host-local loopback] at [\"/opt/cni/bin\" \"/opt/cni/bin\"]

https://github.com/Mirantis/cri-dockerd/releases/tag/v0.2.4 (previously it would use the Docker network, by default)

@afbjorklund
Copy link

And now it is up to "cni" by default, and "no-op" by choice.

https://github.com/Mirantis/cri-dockerd/releases/tag/v0.2.5

@afbjorklund
Copy link

Now the docker removal is complete, since it was not supported in time for the (initial) 1.26 release.

It will probably be done retroactively in 1.27 timeframe, like how CNI was added retroactively with 1.25.
But the skeptics were right, and it would be better to move to the container runtime that is supported ?

Will look to nerdctl for picking up the ball, I would rather not resort to having to use the proprietary tools.

@dims
Copy link
Member

dims commented Dec 10, 2022

It's been exactly a year. I am gonna close this now. please open fresh issues as needed in appropriate repositories.

/close

@k8s-ci-robot
Copy link
Contributor

@dims: Closing this issue.

In response to this:

It's been exactly a year. I am gonna close this now. please open fresh issues as needed in appropriate repositories.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docker lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. sig/docs Categorizes an issue or PR as relevant to SIG Docs. sig/node Categorizes an issue or PR as relevant to SIG Node. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Development

No branches or pull requests