[FEATURE] Replace K3OS

[yasker]

We need to replace K3OS with an OS that can be supported by Rancher/SUSE.

Disclaimer: Somehow this comment was picked up by people and was interpreted in an unintended way. So let me clarify:

1. We're replacing K3OS since K3OS has never been an officially supported OS by either Rancher or SUSE. We need something we can support as the base OS for the Harvester.
2. This change has ABSOLUTELY NOTHING to do with the future of K3S. On the one hand, K3S is a CNCF project, which is not even owned by Rancher/SUSE. On the other hand, K3S is one of the most popular K8s distros in the world, and people love it. There is zero reasons to be concerned about Rancher/SUSE's dedication to the K3S.
3. As you might have seen, Harvester is switching from K3S to RKE2, which is a technology that has a strong connection with K3S. K3S is in fact one of the upstream projects for RKE2. As we mentioned here:

How is this different from RKE or K3s?

RKE2 combines the best-of-both-worlds from the 1.x version of RKE (hereafter referred to as RKE1) and K3s.

From K3s, it inherits the usability, ease-of-operations, and deployment model.

From RKE1, it inherits close alignment with upstream Kubernetes. In places K3s has diverged from upstream Kubernetes in order to optimize for edge deployments, but RKE1 and RKE2 can stay closely aligned with upstream.

Importantly, RKE2 does not rely on Docker as RKE1 does. RKE1 leveraged Docker for deploying and managing the control plane components as well as the container runtime for Kubernetes. RKE2 launches control plane components as static pods, managed by the kubelet. The embedded container runtime is contained.

[chifu1234]

@yasker What do you mean with supported by Rancher/SUSE?

[yasker]

@chifu1234 K3OS is not officially listed as a supported product even for Rancher Labs. It's more like an experimental project at the moment. To support an OS, we need to constantly provide package updates, security updates, etc. So we need to switch to another OS that is supported by Rancher/SUSE.

[belgaied2]

Is it already decided which OS to switch to ? Maybe SLE Micro ?

[bk201]

The OS will be based on RancherOSv2 (A OpenSUSE derivative created by cOS-toolkit). Here are some items to do:

Update: latest items can be found in the description of harvester/harvester-installer#93

[yasker]

Regarding the upgrade path, I've discussed with @ibuildthecloud. Some update here:

1. To upgrade the OS and Harvester, we need a Docker image registry and a Helm repo, even in the air gap environment.
2. The related images and charts are already packaged in the latest ISO.
3. We can add another mode to the installer (as a part of ISO), to serve the Docker image and Helm repo using the content in the ISO.
4. Harvester can start a VM with that ISO, then serve it as the upgrade server to the Harvester cluster
   1. Need to figure out the IP, and maintain HA for the VM, e.g. when upgrading the server VM is running on or have interrupted upgrade due to various reasons.

[bk201]

Thanks for the update.

Regarding the upgrade path, I've discussed with @ibuildthecloud. Some update here:

1. To upgrade the OS and Harvester, we need a Docker image registry and a Helm repo, even in the air gap environment.

Do we want to go this way in day-1 installation? That is, a harvester installation needs Internet connection by default. And if users want to do installation in an air-gapped environment, they need to setup a registry first.

Currently, we are preloading images in the installation stage. And the benefit is user can bring up a cluster very quickly in any environment without registry setup hassle.

2. The related images and charts are already packaged in the latest ISO.
3. We can add another mode to the installer (as a part of ISO), to serve the Docker image and Helm repo using the content in the ISO.

Just to confirm, in this approach a VM (or maybe just a container?) is provisioned in the Live OS?

4. Harvester can start a VM with that ISO, then serve it as the upgrade server to the Harvester cluster

   1. Need to figure out the IP, and maintain HA for the VM, e.g. when upgrading the server VM is running on or have interrupted upgrade due to various reasons.

Maybe we can also use a regular deployment and a service to do this.

[yasker]

@bk201

1. We need to able to support it in the air gap environment, so there won't be a change in day 1 installation. We still want to bring up the cluster with the preloaded images.
2. The VM will be run by Harvester, to serve the Docker image and Helm repo.

[schmitch]

will the new os at least support secure boot? and/or full disk encryption?

[yasker]

@schmitch I think those can be done but are not in the scope for GA. Feel free to file feature requests for them.

[yasker]

OS replacement PR has been merged. All future issues can be tracked by separate tickets.

[mysticaltech]

I think k3os should have been kept, as RancherOS v2 has basically no traction. k3os if moved to k3s-io has a fighting chance!

[DonThorntonJr]

although RancherOS v2 (an immutable Linux distribution...) is a desirable option in a production environment, there is still some development and remediation of the underlying OS for Harvester that can be facilitated through continued and parallel development on K3OS.