Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cluster Provisioning: added RKE2 section #4075

Merged
merged 12 commits into from
May 13, 2022
67 changes: 44 additions & 23 deletions content/rancher/v2.6/en/cluster-provisioning/rke-clusters/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,37 +3,17 @@ title: Launching Kubernetes with Rancher
weight: 4
---

You can have Rancher launch a Kubernetes cluster using any nodes you want. When Rancher deploys Kubernetes onto these nodes, it uses [Rancher Kubernetes Engine]({{<baseurl>}}/rke/latest/en/) (RKE), which is Rancher's own lightweight Kubernetes installer. It can launch Kubernetes on any computers, including:
You can have Rancher launch a Kubernetes cluster using any nodes you want. When Rancher deploys Kubernetes onto these nodes, you can choose between [Rancher Kubernetes Engine]({{<baseurl>}}/rke/latest/en/) (RKE) or [RKE2](https://docs.rke2.io) distributions. Rancher can launch Kubernetes on any computers, including:

- Bare-metal servers
- On-premise virtual machines
- Virtual machines hosted by an infrastructure provider

Rancher can install Kubernetes on existing nodes, or it can dynamically provision nodes in an infrastructure provider and install Kubernetes on them.

RKE clusters include clusters that Rancher launched on Windows nodes or other existing custom nodes, as well as clusters that Rancher launched with new nodes on Azure, Digital Ocean, EC2, or vSphere.
Rancher can also create pools of nodes. One benefit of installing Kubernetes on node pools hosted by an infrastructure provider is that if a node loses connectivity with the cluster, Rancher can automatically create another node to join the cluster to ensure that the count of the node pool is as expected.

### Changes in Rancher v2.6

_Tech Preview_

Rancher v2.6 introduces provisioning for [RKE2](https://docs.rke2.io/) clusters directly from the Rancher UI. RKE2, also known as RKE Government, is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector.

RKE2 provisioning is built on top of a new provisioning framework that leverages the upstream [Cluster API](https://github.com/kubernetes-sigs/cluster-api) project. With this new provisioning framework, you can:

- Provision RKE2 clusters on Digital Ocean, AWS EC2, Azure, and vSphere
- Fully configure RKE2 clusters within Rancher
- Choose CNI options Calico, Cilium, and Multus in addition to Canal
- Install custom RKE2 clusters on pre-provisioned VMs or bare-metal nodes

The RKE2 provisioning tech preview also includes installing RKE2 on Windows clusters. Windows features for RKE2 include:

- Windows Containers with RKE2 powered by containerd
- Added provisioning of Windows RKE2 custom clusters directly from the Rancher UI
- Calico CNI for Windows RKE2 custom clusters.
- SAC releases of Windows Server (2004 and 20H2) are included in the technical preview.

Windows Support for RKE2 Custom Clusters requires choosing Calico as the CNI.
## RKE

### Requirements

Expand All @@ -58,3 +38,44 @@ For more information, refer to the section on [custom nodes.]({{<baseurl>}}/ranc
# Programmatically Creating RKE Clusters

The most common way to programmatically deploy RKE clusters through Rancher is by using the Rancher2 Terraform provider. The documentation for creating clusters with Terraform is [here.](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster)

## RKE2

Rancher v2.6 introduced provisioning for [RKE2](https://docs.rke2.io/) clusters directly from the Rancher UI. RKE2, also known as RKE Government, is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector. In Rancher v.2.6.4 and earlier, RKE2 provisioning was in tech preview.

Note that in Rancher v2.6.5, RKE2 provisioning became GA.

### Requirements

If you use RKE2 to set up a cluster, your nodes must meet the [requirements](https://docs.rke2.io/install/requirements/) for nodes in downstream user clusters.

### Launching Kubernetes on New Nodes in an Infrastructure Provider

RKE2 provisioning is built on top of a new provisioning framework that leverages the upstream [Cluster API](https://github.com/kubernetes-sigs/cluster-api) project. With this new provisioning framework, you can:

- Provision RKE2 clusters onto any provider for which Rancher has a node driver
- Fully configure RKE2 clusters within Rancher
- Choose CNI options Calico, Cilium, and Multus in addition to Canal

nunix marked this conversation as resolved.
Show resolved Hide resolved
RKE2 provisioning also includes installing RKE2 on clusters with Windows nodes.

Windows features for RKE2 include:
jtravee marked this conversation as resolved.
Show resolved Hide resolved

nunix marked this conversation as resolved.
Show resolved Hide resolved
- Windows supports the vSphere node driver
- Calico CNI for Windows RKE2 custom clusters
- Project Network Isolation (PNI) for Calico
- Windows Containers with RKE2 powered by containerd
- Provisioning of Windows RKE2 clusters through Terraform
- Provisioning of Windows RKE2 custom clusters directly from the Rancher UI

Windows Support for RKE2 Custom Clusters requires choosing Calico as the CNI.
jtravee marked this conversation as resolved.
Show resolved Hide resolved

### Launching Kubernetes on Existing Custom Nodes

RKE2 provisioning also allows you to install custom clusters on pre-provisioned VMs or bare-metal nodes.

If you want to reuse a node from a previous custom cluster, clean the node before using it in a cluster again. If you reuse a node that hasn't been cleaned, cluster provisioning may fail.

# Programmatically Creating RKE2 Clusters

The most common way to programmatically deploy RKE2 clusters through Rancher is by using the Rancher2 Terraform provider. The documentation for creating clusters with Terraform is [here.](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster_v2)
Original file line number Diff line number Diff line change
Expand Up @@ -3,58 +3,55 @@ title: Launching Kubernetes on New Nodes in an Infrastructure Provider
weight: 2205
---

Using Rancher, you can create pools of nodes based on a [node template]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/#node-templates). This node template defines the parameters you want to use to launch nodes in your infrastructure providers or cloud providers.

One benefit of installing Kubernetes on node pools hosted by an infrastructure provider is that if a node loses connectivity with the cluster, Rancher can automatically create another node to join the cluster to ensure that the count of the node pool is as expected.

The available cloud providers to create a node template are decided based on active [node drivers]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/#node-drivers).

This section covers the following topics:

- [Changes in Rancher v2.6](#changes-in-rancher-v2-6)
- [Node templates](#node-templates)
- [Node labels](#node-labels)
- [Node taints](#node-taints)
- [Administrator control of node templates](#administrator-control-of-node-templates)
- [Node pools](#node-pools)
- [Node pool taints](#node-pool-taints)
- [About node auto-replace](#about-node-auto-replace)
- [Enabling node auto-replace](#enabling-node-auto-replace)
- [Disabling node auto-replace](#disabling-node-auto-replace)
- [Cloud credentials](#cloud-credentials)
- [Node drivers](#node-drivers)
- [Node roles in RKE2](#node-roles-in-rke2)

# Changes in Rancher v2.6

_Tech Preview_

Rancher v2.6 introduces provisioning for [RKE2](https://docs.rke2.io/) clusters directly from the Rancher UI. RKE2, also known as RKE Government, is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector.
- [RKE Clusters](#rke-clusters)
- [Node templates](#node-templates)
- [Node labels](#node-labels)
- [Node taints](#node-taints)
- [Administrator control of node templates](#administrator-control-of-node-templates)
- [Node pools](#node-pools)
- [Node pool taints](#node-pool-taints)
- [About node auto-replace](#about-node-auto-replace)
- [Enabling node auto-replace](#enabling-node-auto-replace)
- [Disabling node auto-replace](#disabling-node-auto-replace)
- [Cloud credentials](#cloud-credentials)
- [Node drivers](#node-drivers)
- [RKE2 Clusters](#rke2-clusters)
- [Node roles in RKE2](#node-roles-in-rke2)

When you create an RKE or RKE2 cluster using a node template in Rancher, each resulting node pool is shown in a new **Machine Pools** tab. You can see the machine pools by doing the following:

1. Click **☰ > Cluster Management**.
1. Click the name of the RKE or RKE2 cluster.

# Node Templates
## RKE Clusters

Using Rancher, you can create pools of nodes based on a [node template]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/#node-templates). This node template defines the parameters you want to use to launch nodes in your infrastructure providers or cloud providers.

One benefit of installing Kubernetes on node pools hosted by an infrastructure provider is that if a node loses connectivity with the cluster, Rancher can automatically create another node to join the cluster to ensure that the count of the node pool is as expected.

The available cloud providers to create a node template are decided based on active [node drivers]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/#node-drivers).

### Node Templates

A node template is the saved configuration for the parameters to use when provisioning nodes in a specific cloud provider. These nodes can be launched from the UI. Rancher uses [Docker Machine](https://docs.docker.com/machine/) to provision these nodes. The available cloud providers to create node templates are based on the active node drivers in Rancher.

After you create a node template in Rancher, it's saved so that you can use this template again to create node pools. Node templates are bound to your login. After you add a template, you can remove them from your user profile.

### Node Labels
#### Node Labels

You can add [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) on each node template, so that any nodes created from the node template will automatically have these labels on them.

Invalid labels can prevent upgrades or can prevent Rancher from starting. For details on label syntax requirements, see the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set)

### Node Taints
#### Node Taints

You can add [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) on each node template, so that any nodes created from the node template will automatically have these taints on them.

Since taints can be added at a node template and node pool, if there is no conflict with the same key and effect of the taints, all taints will be added to the nodes. If there are taints with the same key and different effect, the taints from the node pool will override the taints from the node template.

### Administrator Control of Node Templates
#### Administrator Control of Node Templates

Administrators can control all node templates. Admins can now maintain all the node templates within Rancher. When a node template owner is no longer using Rancher, the node templates created by them can be managed by administrators so the cluster can continue to be updated and maintained.

Expand All @@ -65,7 +62,7 @@ To access all node templates, an administrator will need to do the following:

**Result:** All node templates are listed. The templates can be edited or cloned by clicking the **⋮**.

# Node Pools
### Node Pools

Using Rancher, you can create pools of nodes based on a [node template](#node-templates).

Expand All @@ -75,31 +72,31 @@ The benefit of using a node pool is that if a node is destroyed or deleted, you

Each node pool must have one or more nodes roles assigned.

Each node role (i.e. etcd, control plane, and worker) should be assigned to a distinct node pool. Although it is possible to assign multiple node roles to a node pool, this should not be done for production clusters.
Each node role (i.e. etcd, controlplane, and worker) should be assigned to a distinct node pool. Although it is possible to assign multiple node roles to a node pool, this should not be done for production clusters.

The recommended setup is to have:

- a node pool with the etcd node role and a count of three
- a node pool with the control plane node role and a count of at least two
- a node pool with the controlplane node role and a count of at least two
- a node pool with the worker node role and a count of at least two

### Node Pool Taints
#### Node Pool Taints

If you haven't defined [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) on your node template, you can add taints for each node pool. The benefit of adding taints at a node pool is beneficial over adding it at a node template is that you can swap out the node templates without worrying if the taint is on the node template.

For each taint, they will automatically be added to any created node in the node pool. Therefore, if you add taints to a node pool that have existing nodes, the taints won't apply to existing nodes in the node pool, but any new node added into the node pool will get the taint.

When there are taints on the node pool and node template, if there is no conflict with the same key and effect of the taints, all taints will be added to the nodes. If there are taints with the same key and different effect, the taints from the node pool will override the taints from the node template.

### About Node Auto-replace
#### About Node Auto-replace

If a node is in a node pool, Rancher can automatically replace unreachable nodes. Rancher will use the existing node template for the given node pool to recreate the node if it becomes inactive for a specified number of minutes.

> **Important:** Self-healing node pools are designed to help you replace worker nodes for <b>stateless</b> applications. It is not recommended to enable node auto-replace on a node pool of master nodes or nodes with persistent volumes attached, because VMs are treated ephemerally. When a node in a node pool loses connectivity with the cluster, its persistent volumes are destroyed, resulting in data loss for stateful applications.

Node auto-replace works on top of the Kubernetes node controller. The node controller periodically checks the status of all the nodes (configurable via the `--node-monitor-period` flag of the `kube-controller`). When a node is unreachable, the node controller will taint that node. When this occurs, Rancher will begin its deletion countdown. You can configure the amount of time Rancher waits to delete the node. If the taint is not removed before the deletion countdown ends, Rancher will proceed to delete the node object. Rancher will then provision a node in accordance with the set quantity of the node pool.

### Enabling Node Auto-replace
#### Enabling Node Auto-replace

When you create the node pool, you can specify the amount of time in minutes that Rancher will wait to replace an unresponsive node.

Expand All @@ -109,7 +106,7 @@ When you create the node pool, you can specify the amount of time in minutes tha

**Result:** Node auto-replace is enabled for the node pool.

### Disabling Node Auto-replace
#### Disabling Node Auto-replace

You can disable node auto-replace from the Rancher UI with the following steps:

Expand All @@ -120,7 +117,7 @@ You can disable node auto-replace from the Rancher UI with the following steps:

**Result:** Node auto-replace is disabled for the node pool.

# Cloud Credentials
### Cloud Credentials

Node templates can use cloud credentials to store credentials for launching nodes in your cloud provider, which has some benefits:

Expand All @@ -132,20 +129,26 @@ Node templates can use cloud credentials to store credentials for launching node

After cloud credentials are created, the user can start [managing the cloud credentials that they created]({{<baseurl>}}/rancher/v2.6/en/user-settings/cloud-credentials/).

# Node Drivers
### Node Drivers

If you don't find the node driver that you want to use, you can see if it is available in Rancher's built-in [node drivers and activate it]({{<baseurl>}}/rancher/v2.6/en/admin-settings/drivers/node-drivers/#activating-deactivating-node-drivers), or you can [add your own custom node driver]({{<baseurl>}}/rancher/v2.6/en/admin-settings/drivers/node-drivers/#adding-custom-node-drivers).

# Node Roles in RKE2
## RKE2 Clusters

The RKE2 CLI exposes two roles, `server` and `agent`, which represent the Kubernetes node-roles `etcd` + `control-plane` and `worker` respectively. With RKE2 integration in Rancher v2.6, RKE2 node pools can represent more fine-grained role assignments such that `etcd` and `control-plane` roles can be represented.
Rancher v2.6 introduces provisioning for [RKE2](https://docs.rke2.io/) clusters directly from the Rancher UI. RKE2, also known as RKE Government, is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector.

> **Note:** For RKE2 cluster templates, please refer to [this page]({{<baseurl>}}/rancher/v2.6/en/admin-settings/cluster-templates/#rke2-cluster-template) for additional information.

### Node Roles

The RKE2 CLI exposes two roles, `server` and `agent`, which represent the Kubernetes node-roles `etcd` + `controlplane` and `worker` respectively. With RKE2 integration in Rancher v2.6, RKE2 node pools can represent more fine-grained role assignments such that `etcd` and `controlplane` roles can be represented.

The same functionality of using `etcd`, `controlplane` and `worker` nodes is possible in the RKE2 CLI by using flags and node tainting to control where workloads and the Kubernetes master were scheduled. The reason those roles were not implemented as first-class roles in the RKE2 CLI is that RKE2 is conceptualized as a set of raw building blocks that are best leveraged through an orchestration system such as Rancher.

The implementation of the three node roles in Rancher means that Rancher managed RKE2 clusters are able to easily leverage all of the same architectural best practices that are recommended for RKE clusters.

In our [recommended cluster architecture]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/production/recommended-architecture/), we outline how many nodes of each role clusters should have:

- At least three nodes with the role etcd to survive losing one node
- At least two nodes with the role controlplane for master component high availability
- At least two nodes with the role worker for workload rescheduling upon node failure

The implementation of the three node roles in Rancher means that Rancher managed RKE2 clusters are able to easily leverage all of the same architectural best practices that are recommended for RKE clusters.
- At least two nodes with the role worker for workload rescheduling upon node failure