Skip to content

Commit

Permalink
Rework section on Fixed Address Registration (#135)
Browse files Browse the repository at this point in the history
* Reword language around fixed registration address
* Bump docusaurus

Signed-off-by: Derek Nola <derek.nola@suse.com>
Co-authored-by: Brad Davidson <brad@oatmail.org>
  • Loading branch information
dereknola and brandond committed Jun 6, 2023
1 parent 2afd3dc commit 2819b88
Show file tree
Hide file tree
Showing 4 changed files with 259 additions and 217 deletions.
19 changes: 15 additions & 4 deletions docs/datastore/ha-embedded.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,15 @@ weight: 40
Embedded etcd (HA) may have performance issues on slower disks such as Raspberry Pis running with SD cards.
:::

## New cluster
To run K3s in this mode, you must have an odd number of server nodes. We recommend starting with three nodes.
:::info
HA embedded etcd cluster must be comprised of an odd number of server nodes for etcd to maintain quorum. For a cluster with n servers, quorum is (n/2)+1. For any odd-sized cluster, adding one node will always increase the number of nodes necessary for quorum. Although adding a node to an odd-sized cluster appears better since there are more machines, the fault tolerance is worse since exactly the same number of nodes may fail without losing quorum but there are more nodes that can fail.
:::

An HA K3s cluster with embedded etcd is composed of:

* Three or more **server nodes** that will serve the Kubernetes API and run other control plane services, as well as host the embedded etcd datastore.
* Optional: Zero or more **agent nodes** that are designated to run your apps and services
* Optional: A **fixed registration address** for agent nodes to register with the cluster

To get started, first launch a server node with the `cluster-init` flag to enable clustering and a token that will be used as a shared secret to join additional servers to the cluster.
```bash
Expand All @@ -30,15 +37,19 @@ server2 Ready control-plane,etcd,master 13m vX.Y.Z
server3 Ready control-plane,etcd,master 10m vX.Y.Z
```

Now you have a highly available control plane. Any successfully clustered servers can be used in the `--server` argument to join additional server and agent nodes. Joining additional agent nodes to the cluster follows the same procedure as a single server cluster.
Now you have a highly available control plane. Any successfully clustered servers can be used in the `--server` argument to join additional server and agent nodes. Joining additional agent nodes to the cluster follows the same procedure as servers:

```bash
curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - agent --server https://<ip or hostname of server>:6443
```

There are a few config flags that must be the same in all server nodes:

* Network related flags: `--cluster-dns`, `--cluster-domain`, `--cluster-cidr`, `--service-cidr`
* Flags controlling the deployment of certain components: `--disable-helm-controller`, `--disable-kube-proxy`, `--disable-network-policy` and any component passed to `--disable`
* Feature related flags: `--secrets-encryption`

## Existing clusters
## Existing single-node clusters
If you have an existing cluster using the default embedded SQLite database, you can convert it to etcd by simply restarting your K3s server with the `--cluster-init` flag. Once you've done that, you'll be able to add additional instances as described above.

If an etcd datastore is found on disk either because that node has either initialized or joined a cluster already, the datastore arguments (`--cluster-init`, `--server`, `--datastore-endpoint`, etc) are ignored.
Expand Down
40 changes: 20 additions & 20 deletions docs/datastore/ha.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,15 @@ weight: 30

This section describes how to install a high-availability K3s cluster with an external database.

Single server clusters can meet a variety of use cases, but for environments where uptime of the Kubernetes control plane is critical, you can run K3s in an HA configuration. An HA K3s cluster is comprised of:
Single server clusters can meet a variety of use cases, but for environments where uptime of the Kubernetes control plane is critical, you can run K3s in an HA configuration. An HA K3s cluster is composed of:

* Two or more **server nodes** that will serve the Kubernetes API and run other control plane services
* Zero or more **agent nodes** that are designated to run your apps and services
* An **external datastore** (as opposed to the embedded SQLite datastore used in single-server setups)
* A **fixed registration address** that is placed in front of the server nodes to allow agent nodes to register with the cluster
* Optional: Zero or more **agent nodes** that are designated to run your apps and services
* Optional: A **fixed registration address** for agent nodes to register with the cluster

For more details on how these components work together, refer to the [architecture section.](../architecture/architecture.md#high-availability-k3s)

Agents register through the fixed registration address, but after registration they establish a connection directly to one of the server nodes. This is a websocket connection initiated by the `k3s agent` process, it is maintained by a client-side load balancer running as part of the agent process.

## Installation Outline

Setting up an HA cluster requires the following steps:
Expand Down Expand Up @@ -48,17 +46,7 @@ By default, server nodes will be schedulable and thus your workloads can get lau

Once you've launched the `k3s server` process on all server nodes, ensure that the cluster has come up properly with `k3s kubectl get nodes`. You should see your server nodes in the Ready state.

### 3. Optional: Configure a Fixed Registration Address

Agent nodes need a URL to register against. This can be the IP or hostname of any of the server nodes, but in many cases those may change over time. For example, if you are running your cluster in a cloud that supports scaling groups, you may scale the server node group up and down over time, causing nodes to be created and destroyed and thus having different IPs from the initial set of server nodes. Therefore, you should have a stable endpoint in front of the server nodes that will not change over time. This endpoint can be set up using any number approaches, such as:

* A layer-4 (TCP) load balancer
* Round-robin DNS
* Virtual or elastic IP addresses

This endpoint can also be used for accessing the Kubernetes API. So you can, for example, modify your [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) file to point to it instead of a specific node. To avoid certificate errors in such a configuration, you should install the server with the `--tls-san YOUR_IP_OR_HOSTNAME_HERE` option. This option adds an additional hostname or IP as a Subject Alternative Name in the TLS cert, and it can be specified multiple times if you would like to access via both the IP and the hostname.

### 4. Optional: Join Additional Server Nodes
### 3. Optional: Join Additional Server Nodes

The same example command in Step 2 can be used to join additional server nodes, where the token from the first node needs to be used.

Expand All @@ -85,12 +73,24 @@ There are a few config flags that must be the same in all server nodes:
Ensure that you retain a copy of this token as it is required when restoring from backup and adding nodes. Previously, K3s did not enforce the use of a token when using external SQL datastores.
:::

### 5. Optional: Join Agent Nodes
### 4. Optional: Join Agent Nodes

Because K3s server nodes are schedulable by default, the minimum number of nodes for an HA K3s server cluster is two server nodes and zero agent nodes. To add nodes designated to run your apps and services, join agent nodes to your cluster.
Because K3s server nodes are schedulable by default, agent nodes are not required for a HA K3s cluster. However, you may wish to have dedicated agent nodes to run your apps and services.

Joining agent nodes in an HA cluster is the same as joining agent nodes in a single server cluster. You just need to specify the URL the agent should register to and the token it should use.
Joining agent nodes in an HA cluster is the same as joining agent nodes in a single server cluster. You just need to specify the URL the agent should register to (either one of the server IPs or a fixed registration address) and the token it should use.

```bash
K3S_TOKEN=SECRET k3s agent --server https://fixed-registration-address:6443
K3S_TOKEN=SECRET k3s agent --server https://server-or-fixed-registration-address:6443
```

### 5. Optional: Configure a Fixed Registration Address

Agent nodes need a URL to register against. This can be the IP or hostname of any server node, but in many cases those may change over time. For example, if you are running your cluster in a cloud that supports scaling groups, you may scale the server node group up and down over time, causing nodes to be created and destroyed and thus having different IPs from the initial set of server nodes. In this case, you should have a stable endpoint in front of the server nodes that will not change over time. This endpoint can be set up using any number approaches, such as:

* A layer-4 (TCP) load balancer
* Round-robin DNS
* Virtual or elastic IP addresses

This endpoint can also be used for accessing the Kubernetes API. So you can, for example, modify your [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) file to point to it instead of a specific node.

To avoid certificate errors in such a configuration, you should configure the server with the `--tls-san YOUR_IP_OR_HOSTNAME_HERE` option. This option adds an additional hostname or IP as a Subject Alternative Name in the TLS cert, and it can be specified multiple times if you would like to access via both the IP and the hostname.
8 changes: 4 additions & 4 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@
"write-heading-ids": "docusaurus write-heading-ids"
},
"dependencies": {
"@docusaurus/core": "^2.3.1",
"@docusaurus/preset-classic": "^2.3.1",
"@docusaurus/theme-mermaid": "^2.3.1",
"@docusaurus/core": "^2.4.0",
"@docusaurus/preset-classic": "^2.4.0",
"@docusaurus/theme-mermaid": "^2.4.0",
"@mdx-js/react": "^1.6.22",
"clsx": "^1.1.1",
"prism-react-renderer": "^1.3.3",
Expand All @@ -29,7 +29,7 @@
"remark-validate-links-heading-id": "^0.0.3"
},
"devDependencies": {
"@docusaurus/module-type-aliases": "^2.3.1"
"@docusaurus/module-type-aliases": "^2.4.0"
},
"browserslist": {
"production": [
Expand Down

0 comments on commit 2819b88

Please sign in to comment.