Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
36 changes: 36 additions & 0 deletions content/operatingsystemmanager/_index.en.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
+++
title = "OperatingSystemManager"
date = 2022-08-20T12:00:00+02:00
weight = 6
description = "Learn how you can use Kubermatic OperatingSystemManager to manage and customize worker node configurations for more granular control over your kubernetes clusters."
+++

![OperatingSystemManager logo](/img/operatingsystemmanager/common/operating-system-manager-logo.png)

## What is OperatingSystemManager?

[Operating System Manager (OSM)][operating-system-manager] is an open source project by Kubermatic, it is responsible for creating and managing the required configurations for worker nodes in a Kubernetes cluster. It decouples operating system configurations into dedicated and isolable resources for better modularity and maintainability.

These isolated and extensible resources allow a high degree of customization which allows users to modify the worker node configurations to suit their use cases. This is useful for hybrid, edge, and air-gapped environments.

Configurations for worker nodes comprise of set of scripts used to prepare the node, install packages, configure networking, storage etc. These configurations prepare the nodes for running `kubelet`.

## Problem Statement

[Machine-Controller][machine-controller] is used to manage the worker nodes in KubeOne clusters. It depends on user-data plugins to generate the required configurations for worker nodes. Each operating system requires its own user-data plugin. These configs are then injected into the worker nodes using provisioning utilities such as [cloud-init](https://cloud-init.io) or [ignition](https://coreos.github.io/ignition). Eventually the nodes are bootstrapped to become a part of a kubernetes cluster.

This has been the norm till [machine-controller v1.54.0](<https://github.com/kubermatic/machine-controller/releases/tag/v1.54.0>) and it works as expected. Although over time, it has been observed that this workflow has certain limitations.

### Machine Controller Limitations

- Machine Controller expects **ALL** the supported user-data plugins to exist and be ready. User might only be interested in a subset of the available operating systems. For example, user might only want to work with `ubuntu`.
- The user-data plugins have templates defined [in-code](https://github.com/kubermatic/machine-controller/blob/v1.54.0/pkg/userdata/ubuntu/provider.go#L136). Which is not ideal since code changes are required to update those templates. Then those code changes need to become a part of the subsequent releases for machine-controller and KubeOne. So we need a complete release cycle to ship those changes to customers.
- Managing configs for multiple cloud providers, OS flavors and OS versions, adds a lot of complexity and redundancy in machine-controller.
- Since the templates are defined in-code, there is no way for an end user to customize them to suit their use-cases.
- Each cloud provider sets some sort of limits for the size of `user-data`, machine won't be created in case of non-compliance. For example, at the time of writing this, AWS has set a [hard limit of 16KB](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-add-user-data.html).
- Better support for air-gapped environments is required.

Operating System Manager was created to overcome these limitations.

[machine-controller]: https://github.com/kubermatic/machine-controller
[operating-system-manager]: https://github.com/kubermatic/operating-system-manager
49 changes: 49 additions & 0 deletions content/operatingsystemmanager/architecture/_index.en.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
+++
title = "Architecture"
description = "Learn about the architecture of Kubermatic OperatingSystemManager"
date = 2022-08-20T12:00:00+02:00
weight = 6
+++

## Architecture

With an aim to make machine configurations extendable, OSM introduces the following new resources as [Kubernetes Custom Resource Definitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/):

### OperatingSystemProfile

A resource that contains scripts for bootstrapping and provisioning the worker nodes, along with information about what operating systems and versions are supported for given scripts. Additionally, OSPs support templating so you can include some information from MachineDeployment or the OSM deployment itself.

OSM comes with a set of default OSPs for various supported operating systems and cloud provider combinations. These OSPs are installed by OSM itself. End users can create custom OSPs as well to fit their own use-cases.

OSPs are immutable by design and any modifications to an existing OSP requires a version bump in `.spec.version`.

### OperatingSystemConfig

Immutable resource that contains the **actual configurations** that are going to be used to bootstrap and provision the worker nodes. It is a subset of OperatingSystemProfile. OperatingSystemProfile is a template while OperatingSystemConfig is an instance rendered with data from OperatingSystemProfile, MachineDeployment, and flags provided at OSM command-line level.

OperatingSystemConfigs have a 1-to-1 relation with the MachineDeployment. A dedicated controller watches the MachineDeployments and generates the OSCs in `kube-system`(can be overridden).

### Kubernetes Secrets

The generated OperatingSystemConfigs are then transformed into configurations accepted by cloud-init or ignition provisioning utilities. They are persisted as [kubernetes secrets](https://kubernetes.io/docs/concepts/configuration/secret/) in `cloud-init-settings` namespace in the cluster.

The following secrets are generated by OSM:

1. **Bootstrap**: Configuration used for initially setting up the machine and fetching the provisioning configuration.
2. **Provisioning**: Configuration with the actual `cloud-config` that is used to provision the worker machine.

The significance behind using a dedicated namespace i.e. `cloud-init-settings` is that the secret that holds the `provisioning` configuration is dynamically fetched during the `bootstrap` phase. Using a dedicated namespace isolates these secrets and limits the access area.

![Architecture](/img/operatingsystemmanager/master/osm-kubernetes.png?classes=shadow,border "Integration with Machine Controller")

## Single vs management/worker cluster mode

Conventionally OSM operates within a single cluster and expects all of the required resources like machine controller, MachineDeployments etc. to exist within the same cluster.

Along with that, OSM also supports environments where workloads are divided into management and worker clusters. This is useful since it helps with completely abstracting away OSM from the users of worker cluster; OSM will be running in the management cluster.

To use management/worker cluster mode, simply pass on the kubeconfig for management cluster using `kubeconfig` and worker cluster using the `worker-cluster-kubeconfig` flags at OSM level. With this topology the OSP and OSC exist within the management cluster while only the bootstrap and provisioning secrets are created in the worker clusters.

## Integration with machine-controller

When a MachineDeployment is created, OSM reconciles it and generates the corresponding OSC and secrets based on the selected OSP. The OSP to be used for a MachineDeployment is specified using `k8c.io/operating-system-profile` annotation on the MachineDeployment. Machine controller waits for the bootstrapping and provisioning secrets to become available. Once they are made available by OSM, machine controller will extract the configurations from those secrets and propagate them as `user-data` for the machines that are to be provisioned.
8 changes: 8 additions & 0 deletions content/operatingsystemmanager/cheat-sheets/_index.en.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
+++
title = "Cheat Sheets"
date = 2022-08-20T12:00:00+02:00
weight = 7
chapter = true
+++

# Cheat Sheets
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
+++
title = "Rolling Restart MachineDeploments"
date = 2022-08-20T12:00:00+02:00
description = "Rolling restart machine deployments: Learn about the concept and keep it handy"
weight = 25
+++

## To rolling restart a single machineDeployment

```shell
forceRestartAnnotations="{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"forceRestart\":\"$(date +%s)\"}}}}}"
kubectl patch machinedeployment -n kube-system <NAME> --type=merge -p $forceRestartAnnotations
```

## To rolling restart all machineDeployments

```shell
forceRestartAnnotations="{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"forceRestart\":\"$(date +%s)\"}}}}}"
for md in $(kubectl get machinedeployments -n kube-system --no-headers | awk '{print $1}'); do
kubectl patch machinedeployment -n kube-system $md --type=merge -p $forceRestartAnnotations
done
```
31 changes: 31 additions & 0 deletions content/operatingsystemmanager/compatibility/_index.en.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
+++
title = "Compatibility Matrix"
date = 2022-08-20T12:00:00+02:00
weight = 6
+++

The page provides an overview for the supported operating systems on various cloud providers. These are the combinations that are covered by the "default" OperatingSystemProfiles that OSM will install in your cluster. Users can create custom OperatingSystemProfiles that work with a provider/OS combination that are not listed here.

## Operating System

| | Ubuntu | CentOS | Flatcar | Amazon Linux 2 | RHEL | Rocky Linux |
|---|---|---|---|---|---|---|
| AWS | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Azure | ✓ | ✓ | ✓ | x | ✓ | ✓ |
| DigitalOcean | ✓ | ✓ | x | x | x | ✓ |
| Equinix Metal | ✓ | ✓ | ✓ | x | x | ✓ |
| Google Cloud Platform | ✓ | x | x | x | x | x |
| Hetzner | ✓ | ✓ | x | x | x | ✓ |
| KubeVirt | ✓ | ✓ | ✓ | x | ✓ | ✓ |
| Nutanix | ✓ | ✓ | x | x | x | x |
| Openstack | ✓ | ✓ | ✓ | x | ✓ | ✓ |
| VMware Cloud Director | ✓ | x | x | x | x | x |
| VSphere | ✓ | ✓ | ✓ | x | ✓ | ✓ |

## Kubernetes Versions

Currently supported K8S versions are:

- 1.24
- 1.23
- 1.22
Loading