kubeadm cri installation instructions #10186

vincepri · 2018-09-04T20:08:38Z

Fixes kubernetes/kubeadm#1086
Fixes #9692

k8s-ci-robot · 2018-09-04T20:08:46Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: zacharysarah

If they are not already assigned, you can assign the PR to them by writing /assign @zacharysarah in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8sio-netlify-preview-bot · 2018-09-04T20:13:05Z

Deploy preview for kubernetes-io-master-staging ready!

Built with commit 7876861

https://deploy-preview-10186--kubernetes-io-master-staging.netlify.com

k8sio-netlify-preview-bot · 2018-09-04T20:13:41Z

Deploy preview for kubernetes-io-master-staging ready!

Built with commit 22ca1d1

https://deploy-preview-10186--kubernetes-io-master-staging.netlify.com

timothysc · 2018-09-04T20:13:50Z

/cc @kubernetes/sig-cluster-lifecycle-pr-reviews @kubernetes/sig-node-pr-reviews

detiber · 2018-09-04T20:22:26Z

content/en/docs/reference/setup-tools/kubeadm/cri-installation.md

+echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
+echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
+echo 1 > /proc/sys/net/ipv4/ip_forward
+sysctl -p


Should we be telling users to modify /etc/sysctl.conf instead of using echo? Otherwise sysctl -p doesn't really have any effect here, and the changes will not persist through a reboot.

Good point. Maybe we can add a file to /etc/sysctl.d/? Something like 99-kubernetes-cri.conf should do the trick.

detiber · 2018-09-04T20:23:48Z

content/en/docs/reference/setup-tools/kubeadm/cri-installation.md

+systemctl start containerd
+
+# Setup kubelet to use containerd runtime endpoint and reload systemd.
+cat > /etc/systemd/system/kubelet.service.d/20-cri.conf <<EOF


We really shouldn't be telling users to override the unit in this way, instead they should be passing this info in to the kubeadm config.

@detiber Something like

nodeRegistration: kubeletExtraArgs: container-runtime-endpoint: unix:///var/run/containerd/containerd.sock

would work?

neolit123

@vincepri
thanks a lot for the PR!
added some minor comments.

neolit123 · 2018-09-04T20:16:08Z

content/en/docs/reference/setup-tools/kubeadm/cri-installation.md

@@ -0,0 +1,144 @@
+---


we should move this doc to somewhere else.
reference/setup-tools/kubeadm/ is the folder about the kubeadm CLI flags docs.
and move the kubeadm init content too.

my initial suggestion would be:
-> docs/setup/independent
if this guide is kubeadm related.

if not it has to go somewhere else.

Agreed. The goal is to have this as a general CRI installation document and reference it as needed. Where would you suggest to have it reside?

the structure of the k8s.io documentation is bit confusing.
we have both tasks and setup and the answer to the question is difficult.

i would add this under a new folder under docs/setup/ called cri (title: Kubernetes CRI):

the folder needs an _index.md as this one to set the title:
https://github.com/kubernetes/website/tree/master/content/en/docs/setup/independent

the name cri-installation.md as a file under that folder SGTM.

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

content/en/docs/reference/setup-tools/kubeadm/cri-installation.md

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

neolit123 · 2018-09-04T21:38:41Z

content/en/docs/reference/setup-tools/kubeadm/cri-installation.md

+---
+reviewers:
+- vincepri
+title: CRI Installation


please lowercase Installation i think we should have this consistent for other pages too.

neolit123 · 2018-09-04T21:54:38Z

thank you for the update @vincepri
let's see if others have more comments.

vincepri · 2018-09-04T21:57:22Z

@neolit123 @detiber Thank you for the feedback! I addressed all the comments. Let me know if there is anything else.

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

Signed-off-by: Vince Prignano <vince@vincepri.com>

neolit123 · 2018-09-05T17:27:14Z

@vincepri
thanks for the update removing the docker instructions from the general kubeadm setup guide.
LGTM

timothysc · 2018-09-05T19:21:05Z

content/en/docs/setup/independent/install-kubeadm.md

-
-Refer to the [official Docker installation guides](https://docs.docker.com/engine/installation/)
-for more information.
+Refer to the [CRI installation](/docs/setup/cri/cri-installation/) guide for more information.


We will need to outline that Docker is still the default and if another CRI and outline the command line overrides in the kubeadm init / join below.

You might say something like:

By default, kubernetes is configured to work with docker (18.06). In order to enable other CRIs please consult https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/ and https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/ instructions.

Is the docker version relevant? In the CRI docs instructions we're still suggesting to install 17.03, should we update it? This is especially relevant for Ubuntu 18.04.

so that's a good question.
17.03 is the last verified version by SIG node and as Tim mentioned yesterday they are not going to verify a newer version.

these docs here seem out of date:
https://docs.docker.com/release-notes/docker-ce/

i think this shows the truth:
https://github.com/docker/docker-ce/releases
18.06.1-ce - latest edge
18.03.1-ce - latest stable

I think we should include the information you just shared in the CRI document, under the Docker section. The install-kubeadm one can be left version-less?

it really depends of what version is tested with k8s, because we cannot recommend even a stable docker version if it's broken with k8s latest.
i will defer to others for more comments.

@timothysc thoughts/preferences on the above?

@vincepri
PSA, we are soon going to merge a PR that updates the max validated version of docker for kubeadm to 18.06:
kubernetes/kubernetes#68495

we need to reflect this in the docs as recommended version.
at least attempt to, given all the distro flavors that we have to support.

timothysc · 2018-09-09T19:50:47Z

/assign @Bradamant3

bartosh · 2018-09-10T09:16:25Z

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

-Since v1.6.0, Kubernetes has enabled the use of CRI, Container Runtime Interface, by default.
-The container runtime used by default is Docker, which is enabled through the built-in
-`dockershim` CRI implementation inside of the `kubelet`.
+From v1.12.0 the suggested kubeadm CRI is containerd. For further information refer to [CRI Installation](/docs/setup/cri/cri-installation/) instructions.


Can you elaborate on this a bit? Why containerd is suggested kubeadm CRI? Why not CRI-O or both?

I believe this was due to the lack of tests with docker, not totally sure about CRI-O. I'll defer to @timothysc.

@neolit123 once we have CI-Signal back we need to double check the versions.

my vote would be to not go with this statement for 1.12.0.
docker is currently the CRI for most of kubeadm users and also all of the kubeadm e2e here are docker too:
https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-all

I agree with @neolit123 here. Let's not promote switching to other runtimes just yet. Let's provide more information on how to set up CRI runtimes and how to use them. That should be enough for 1.12 I believe.

we are pending on decision for this.
i don't have strong opinions here.

the state of tests from sig-node for containerd are yellow-ish:
https://k8s-testgrid.appspot.com/sig-node-containerd

we do not have any tests for kubeadm and containerd yet, but our docker tests are passing at least.

xref:
https://github.com/kubernetes/website/pull/10186/files#r215439277

bart0sh · 2018-09-10T12:31:15Z

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

-[Service]
-Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=$RUNTIME_ENDPOINT"
-EOF
-systemctl daemon-reload


Instructions to update systemd config was quite useful. I'd suggest not to remove it.

Given that this is now covered by kubeadm directly, I think it's safe to remove. The kubelet systemd example should live under kubelet docs.

Current documentation uses both ways of configuring kubelet - using configuration file and systemd drop-ins. In my opinion removing this can create confusion among users who prefer to use latter approach.

bart0sh · 2018-09-10T12:33:12Z

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

@@ -249,7 +249,7 @@ networking:
  podSubnet: ""
  serviceSubnet: 10.96.0.0/12
 nodeRegistration:
-  criSocket: /var/run/dockershim.sock
+  criSocket: /var/run/containerd/containerd.sock


I'd leave this as is as docker is still a default runtime.

Sounds good, thanks!

bart0sh · 2018-09-10T12:34:13Z

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

@@ -368,41 +368,22 @@ Here's a breakdown of what/why:
   certificates from the `kube-apiserver` when the certificate expiration approaches.
 * `--cert-dir`the directory where the TLS certs are located.

-### Use kubeadm with other CRI runtimes
+### Use kubeadm with containerd


I'd propose to leave this as it is and add setup instructions for at least containerd and CRI-O.

looked at rktlet and frakti. Not sure I understand why to remove them from this document.

We could add instructions for other runtimes with kubeadm to this page later on, does that sound good? We should definitely track these.

I think it would be better to leave links to other runtimes documentation even if we don't have our instructions for those. People at least would know that those runtimes exist and they can be also potentially used.

timothysc · 2018-09-10T21:10:32Z

Let's get the CI signal back, double check versions and loop back to this once the mains are back online.

mythi · 2018-09-11T08:57:36Z

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

+After installing containerd, you should set `--cri-socket` in kubeadm init and kubeadm reset. Or, in alternative to command line flags, supply the containerd socket in your kubeadm configuration as shown in the example below:
+```yaml
+nodeRegistration:
+  criSocket: /var/run/containerd/containerd.sock


I did a fresh pull of containerd and the socket path I get is /run/containerd/containerd.sock

bart0sh · 2018-09-14T13:30:21Z

@vincepri Here is an example of what I meant: #10299

The idea is to provide instructions for the CRI runtimes that we can create without changing current documentation too much.

@ALL Does this make sense?

vincepri · 2018-09-14T23:04:40Z

Closing in favor of #10299

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Sep 4, 2018

k8s-ci-robot requested review from mikedanese and zhangxiaoyu-zidif September 4, 2018 20:08

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 4, 2018

k8s-ci-robot requested a review from a team September 4, 2018 20:13

k8s-ci-robot added sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Sep 4, 2018

detiber reviewed Sep 4, 2018

View reviewed changes

neolit123 reviewed Sep 4, 2018

View reviewed changes

vincepri force-pushed the cri-installation branch from 879ccaa to b5500e5 Compare September 4, 2018 21:47

vincepri force-pushed the cri-installation branch from 7603ffb to 8aa38ff Compare September 4, 2018 22:01

feiskyer reviewed Sep 5, 2018

View reviewed changes

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md Outdated Show resolved Hide resolved

vincepri force-pushed the cri-installation branch from 8aa38ff to 5a295bc Compare September 5, 2018 15:43

fabriziopandini reviewed Sep 5, 2018

View reviewed changes

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md Outdated Show resolved Hide resolved

vincepri force-pushed the cri-installation branch from 5a295bc to 75c68a4 Compare September 5, 2018 16:06

Add CRI installation instructions page with containerd and docker

22ca1d1

Signed-off-by: Vince Prignano <vince@vincepri.com>

vincepri force-pushed the cri-installation branch from 11ba78a to 22ca1d1 Compare September 5, 2018 16:24

bart0sh mentioned this pull request Sep 5, 2018

CRI Installation Instructions kubernetes/kubeadm#1086

Closed

vincepri changed the title ~~WIP: kubeadm cri installation instructions~~ kubeadm cri installation instructions Sep 5, 2018

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 5, 2018

timothysc reviewed Sep 5, 2018

View reviewed changes

k8s-ci-robot assigned Bradamant3 Sep 9, 2018

bartosh reviewed Sep 10, 2018

View reviewed changes

bart0sh reviewed Sep 10, 2018

View reviewed changes

mythi reviewed Sep 11, 2018

View reviewed changes

vincepri closed this Sep 14, 2018

vincepri mentioned this pull request Sep 19, 2018

REQUEST: New membership for vincepri kubernetes/org#102

Closed

6 tasks

kubeadm cri installation instructions #10186

kubeadm cri installation instructions #10186

Conversation

vincepri commented Sep 4, 2018 • edited

k8s-ci-robot commented Sep 4, 2018

k8sio-netlify-preview-bot commented Sep 4, 2018

k8sio-netlify-preview-bot commented Sep 4, 2018 • edited

timothysc commented Sep 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vincepri Sep 4, 2018 • edited

Choose a reason for hiding this comment

neolit123 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neolit123 commented Sep 4, 2018

vincepri commented Sep 4, 2018

neolit123 commented Sep 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neolit123 Sep 6, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothysc commented Sep 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothysc commented Sep 10, 2018

Choose a reason for hiding this comment

bart0sh commented Sep 14, 2018

vincepri commented Sep 14, 2018

vincepri commented Sep 4, 2018 •

edited

k8sio-netlify-preview-bot commented Sep 4, 2018 •

edited

vincepri Sep 4, 2018 •

edited

neolit123 Sep 6, 2018 •

edited