Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubeadm cri installation instructions #10186

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 13 additions & 32 deletions content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,7 @@ networking:
podSubnet: ""
serviceSubnet: 10.96.0.0/12
nodeRegistration:
criSocket: /var/run/dockershim.sock
criSocket: /var/run/containerd/containerd.sock
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd leave this as is as docker is still a default runtime.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, thanks!

name: your-host-name
taints:
- effect: NoSchedule
Expand Down Expand Up @@ -368,41 +368,22 @@ Here's a breakdown of what/why:
certificates from the `kube-apiserver` when the certificate expiration approaches.
* `--cert-dir`the directory where the TLS certs are located.

### Use kubeadm with other CRI runtimes
### Use kubeadm with containerd
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd propose to leave this as it is and add setup instructions for at least containerd and CRI-O.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looked at rktlet and frakti. Not sure I understand why to remove them from this document.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could add instructions for other runtimes with kubeadm to this page later on, does that sound good? We should definitely track these.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to leave links to other runtimes documentation even if we don't have our instructions for those. People at least would know that those runtimes exist and they can be also potentially used.


Since v1.6.0, Kubernetes has enabled the use of CRI, Container Runtime Interface, by default.
The container runtime used by default is Docker, which is enabled through the built-in
`dockershim` CRI implementation inside of the `kubelet`.
From v1.12.0 the suggested kubeadm CRI is containerd. For further information refer to [CRI Installation](/docs/setup/cri/cri-installation/) instructions.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate on this a bit? Why containerd is suggested kubeadm CRI? Why not CRI-O or both?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this was due to the lack of tests with docker, not totally sure about CRI-O. I'll defer to @timothysc.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@neolit123 once we have CI-Signal back we need to double check the versions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my vote would be to not go with this statement for 1.12.0.
docker is currently the CRI for most of kubeadm users and also all of the kubeadm e2e here are docker too:
https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-all

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @neolit123 here. Let's not promote switching to other runtimes just yet. Let's provide more information on how to set up CRI runtimes and how to use them. That should be enough for 1.12 I believe.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are pending on decision for this.
i don't have strong opinions here.

the state of tests from sig-node for containerd are yellow-ish:
https://k8s-testgrid.appspot.com/sig-node-containerd

we do not have any tests for kubeadm and containerd yet, but our docker tests are passing at least.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Other CRI-based runtimes include:

- [cri-containerd](https://github.com/containerd/cri-containerd)
- [cri-o](https://github.com/kubernetes-incubator/cri-o)
- [frakti](https://github.com/kubernetes/frakti)
- [rkt](https://github.com/kubernetes-incubator/rktlet)

After you have successfully installed `kubeadm` and `kubelet`, execute
these two additional steps:

1. Install the runtime shim on every node, following the installation
document in the runtime shim project listing above.

1. Configure kubelet to use the remote CRI runtime. Please remember to change
`RUNTIME_ENDPOINT` to your own value like `/var/run/{your_runtime}.sock`:

```shell
cat > /etc/systemd/system/kubelet.service.d/20-cri.conf <<EOF
[Service]
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=$RUNTIME_ENDPOINT"
EOF
systemctl daemon-reload
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instructions to update systemd config was quite useful. I'd suggest not to remove it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that this is now covered by kubeadm directly, I think it's safe to remove. The kubelet systemd example should live under kubelet docs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current documentation uses both ways of configuring kubelet - using configuration file and systemd drop-ins. In my opinion removing this can create confusion among users who prefer to use latter approach.

After installing containerd, you should set `--cri-socket` in kubeadm init and kubeadm reset. Or, in alternative to command line flags, supply the containerd socket in your kubeadm configuration as shown in the example below:
```yaml
nodeRegistration:
criSocket: /var/run/containerd/containerd.sock
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a fresh pull of containerd and the socket path I get is /run/containerd/containerd.sock

```

Now `kubelet` is ready to use the specified CRI runtime, and you can continue
with the `kubeadm init` and `kubeadm join` workflow to deploy Kubernetes cluster.

You may also want to set `--cri-socket` to `kubeadm init` and `kubeadm reset` when
using an external CRI implementation.
In addition, you should set kubectl flag `--container-runtime-endpoint` to containerd socket address. This can be done through kubeadm configuration as shown in the example below:
```yaml
nodeRegistration:
kubeletExtraArgs:
container-runtime-endpoint: unix:///var/run/containerd/containerd.sock
```

### Using internal IPs in your cluster

Expand Down
4 changes: 4 additions & 0 deletions content/en/docs/setup/cri/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: Kubernetes CRI
weight: 30
---
140 changes: 140 additions & 0 deletions content/en/docs/setup/cri/cri-installation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
---
reviewers:
- vincepri
title: CRI installation
content_template: templates/concept
weight: 100
---
{{% capture overview %}}
Since v1.6.0, Kubernetes has enabled the use of CRI, Container Runtime Interface, by default.
This page contains installation instruction for various runtimes.

{{% /capture %}}

{{% capture body %}}

### Containerd

This section contains the necessary steps to use `containerd` as CRI with kubeadm.

#### Prerequisites

```shell
modprobe overlay
modprobe br_netfilter

# Setup required sysctl params, these persist across reboots.
cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl --system
```

{{< tabs name="tab-cri-containerd-installation" >}}
{{< tab name="Ubuntu 16.04+" codelang="bash" >}}
apt-get install -y libseccomp2
{{< /tab >}}
{{< tab name="CentOS/RHEL 7.4+" codelang="bash" >}}
yum install -y libseccomp
{{< /tab >}}
{{< /tabs >}}

#### Install containerd

[Containerd releases](https://github.com/containerd/containerd/releases) are published regularly, the values below are hardcoded to the latest version available at the time of writing. Please check for newer versions and hashes [here](https://storage.googleapis.com/cri-containerd-release).

```shell
# Export required environment variables.
export CONTAINERD_VERSION="1.1.2"
export CONTAINERD_SHA256="d4ed54891e90a5d1a45e3e96464e2e8a4770cd380c21285ef5c9895c40549218"

# Download containerd tar.
wget https://storage.googleapis.com/cri-containerd-release/cri-containerd-${CONTAINERD_VERSION}.linux-amd64.tar.gz

# Check hash.
echo "${CONTAINERD_SHA256} cri-containerd-${CONTAINERD_VERSION}.linux-amd64.tar.gz" | sha256sum --check -

# Unpack.
tar --no-overwrite-dir -C / -xzf cri-containerd-${CONTAINERD_VERSION}.linux-amd64.tar.gz

# Start containerd.
systemctl start containerd
```

### Docker

{{< tabs name="tab-cri-docker-installation" >}}
{{< tab name="Ubuntu 16.04" codelang="bash" >}}
# Install prerequisites.
apt-get install apt-transport-https ca-certificates curl software-properties-common

# Download GPG key.
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

# Add docker apt repository.
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"

# Install docker.
apt-get update && apt-get install docker-ce=17.03.2~ce-0~ubuntu-xenial

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker
{{< /tab >}}
{{< tab name="CentOS/RHEL 7.4+" codelang="bash" >}}
# Install prerequisites.
yum install yum-utils device-mapper-persistent-data lvm2

# Add docker repository.
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo

# Install docker.
apt-get update && yum install docker-ce-17.03.2.ce

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker
{{< /tab >}}
{{< /tabs >}}


{{% /capture %}}
52 changes: 2 additions & 50 deletions content/en/docs/setup/independent/install-kubeadm.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,57 +79,9 @@ The pod network plugin you use (see below) may also require certain ports to be
open. Since this differs with each pod network plugin, please see the
documentation for the plugins about what port(s) those need.

## Installing Docker
## Installing a CRI

On each of your machines, install Docker.
Version 17.03 is recommended, but 1.11, 1.12 and 1.13 are known to work as well.
Versions 17.06+ _might work_, but have not yet been tested and verified by the Kubernetes node team.
Keep track of the latest verified Docker version in the Kubernetes release notes.

Please proceed with executing the following commands based on your OS as root. You may become the root user by executing `sudo -i` after SSH-ing to each host.

If you already have the required versions of the Docker installed, you can move on to next section.
If not, you can use the following commands to install Docker on your system:

{{< tabs name="docker_install" >}}
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
Install Docker from Ubuntu's repositories:

```bash
apt-get update
apt-get install -y docker.io
```

or install Docker CE 17.03 from Docker's repositories for Ubuntu or Debian:

```bash
apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
apt-get update && apt-get install -y docker-ce=$(apt-cache madison docker-ce | grep 17.03 | head -1 | awk '{print $3}')
```
{{% /tab %}}
{{% tab name="CentOS, RHEL or Fedora" %}}
Install Docker using your operating system's bundled package:

```bash
yum install -y docker
systemctl enable docker && systemctl start docker
```
{{% /tab %}}
{{% tab name="Container Linux" %}}
Enable and start Docker:

```bash
systemctl enable docker && systemctl start docker
```
{{% /tab %}}
{{< /tabs >}}


Refer to the [official Docker installation guides](https://docs.docker.com/engine/installation/)
for more information.
Refer to the [CRI installation](/docs/setup/cri/cri-installation/) guide for more information.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need to outline that Docker is still the default and if another CRI and outline the command line overrides in the kubeadm init / join below.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might say something like:

By default, kubernetes is configured to work with docker (18.06). In order to enable other CRIs please consult https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/ and https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/ instructions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the docker version relevant? In the CRI docs instructions we're still suggesting to install 17.03, should we update it? This is especially relevant for Ubuntu 18.04.

Copy link
Member

@neolit123 neolit123 Sep 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so that's a good question.
17.03 is the last verified version by SIG node and as Tim mentioned yesterday they are not going to verify a newer version.

these docs here seem out of date:
https://docs.docker.com/release-notes/docker-ce/

i think this shows the truth:
https://github.com/docker/docker-ce/releases
18.06.1-ce - latest edge
18.03.1-ce - latest stable

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should include the information you just shared in the CRI document, under the Docker section. The install-kubeadm one can be left version-less?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it really depends of what version is tested with k8s, because we cannot recommend even a stable docker version if it's broken with k8s latest.
i will defer to others for more comments.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@timothysc thoughts/preferences on the above?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vincepri
PSA, we are soon going to merge a PR that updates the max validated version of docker for kubeadm to 18.06:
kubernetes/kubernetes#68495

we need to reflect this in the docs as recommended version.
at least attempt to, given all the distro flavors that we have to support.


## Installing kubeadm, kubelet and kubectl

Expand Down