-
Notifications
You must be signed in to change notification settings - Fork 450
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
WINC-505: Windows containerd runtime enablement
Enhancement proposal for making containerd as default runtime in windows node. Signed-off-by: selansen <esiva@redhat.com>
- Loading branch information
selansen
committed
Nov 29, 2021
1 parent
05f2817
commit 05dfc98
Showing
1 changed file
with
213 additions
and
0 deletions.
There are no files selected for viewing
213 changes: 213 additions & 0 deletions
213
enhancements/windows-containers/container-runtime-containerd.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,213 @@ | ||
| --- | ||
| title: container-runtime-containerd | ||
| authors: | ||
| - "@selansen" | ||
| reviewers: | ||
| - "@aravindhp" | ||
| - "@openshift/openshift-team-windows-containers" | ||
| approvers: | ||
| - "@aravindhp" | ||
| - "@mrunalp" | ||
| creation-date: 2021-11-19 | ||
| last-updated: 2021-11-20 | ||
| status: implementable | ||
| --- | ||
|
|
||
| # containerd - new container runtime | ||
|
|
||
| ## Release Signoff Checklist | ||
|
|
||
| - [x] Enhancement is `implementable` | ||
| - [x] Design details are appropriately documented from clear requirements | ||
| - [x] Test plan is defined | ||
| - [x] Graduation criteria for dev preview, tech preview, GA | ||
| - [ ] User-facing documentation is created in [OpenShift-docs](https://github.com/OpenShift/OpenShift-docs/) | ||
|
|
||
| ## Summary | ||
|
|
||
| This enhancement intends to allow customers to bring up Windows node with containerd | ||
| as default runtime from kubernetes 1.24 based OpenShift 4.11 onwards. When customers try to upgrade | ||
| operational cluster to kubernetes 1.24 based OpenShift 4.11, runtime will be migrated from docker to containerd | ||
|
|
||
| ## Motivation | ||
|
|
||
| In Kubernetes, the CRI interface is used to talk to a container runtime. The design of CRI is to be able | ||
| to run a CRI implementation as a separate binary, However currently Docker CRI (a.k.a. dockershim) | ||
| is part of kubelet code, runs as part of kubelet and is tightly coupled with kubelet's lifecycle. From kubernetes | ||
| 1.24 onwards dockershim will be removed from kubelet code. At present WMCO uses Docker CRI as the default runtime. | ||
| Aim is to make containerd as default runtime and move away from Docker CRI before dokershim has been decoupled from | ||
| kubelet. | ||
|
|
||
| ### Goals | ||
|
|
||
| As part of this enhancement we plan to do the following: | ||
| * Make containerd as default runtime. | ||
| * When upgrade happens from WMCO 5.0 to WMCO 6.0, containerd will become default runtime. | ||
|
|
||
| ### Non-Goals | ||
|
|
||
| * De-configuring docker runtime is not part of this enhancement. | ||
| * Refactoring or re-design of WMCO due to dockershim deprecation. | ||
| * Allowing the user to choose container runtime with user facing option is not supported. | ||
| * Downgrade is not supported. | ||
|
|
||
| ## Proposal | ||
|
|
||
| To make containerd as default runtime, containerd should be installed before kueblet and | ||
| kubelet parameters need to be updated so that containerd will become default runtime. In | ||
| upgrade case two scenarios need to be handled. | ||
| 1) MachineSet | ||
| Each Windows VM configured by previous versions of the WMCO has its Machine | ||
| object deleted, resulting in the drain and deletion of the Windows node, and the termination | ||
| then followed by a recreation of the VM. The upgraded WMCO instance will then be able to configure | ||
| VMs that will be created with containerd as default runtime. All minimum requirements stated | ||
| as part of WMCO upgrade will be applicable here as well. The golden image that we used to create | ||
| Windows VM doesn't need to have docker runtime. It is optional as customers may use docker for | ||
| non-kubernetes requirements. | ||
| 2) BYOH ( Bring Your Own Host) | ||
| If Windows VM was created by BYOH option, the upgrade process will uninstall all components | ||
| that were installed by WMCO. As part of the de-configuring process, Windows OS specific configurations | ||
| like HNS-Network that were created by WMCO would have been deleted. During reinstall workflow, containerd | ||
| will be installed along with kubelet, kube-proxy, CNI and hybrid-overlay. | ||
|
|
||
| There is no difference between MachineSet and BYOH on how we enable containerd as default runtime. | ||
|
|
||
| Containerd Migration plan | ||
| *Containerd will become default runtime as part of OpenShift 4.11 | ||
| *For current usage, we plan to introduce WMCO feature flag and release it only for community operator. | ||
| *The default value will be Docker CRI. In 4.10 community branch, the default value will be containerd. | ||
| *Customers will not have the option to choose runtime by using WMCO feature flag. | ||
| *In BYOH case Windows VM Will be de-configured and re-configured with the WMCO supported runtime. | ||
| *In MachineSet option Docker CRI based VM will be deleted and a new VM will be created with containerd | ||
| as default runtime. | ||
|
|
||
| ### User Stories | ||
|
|
||
| Stories can be found within the [Windows Containers: containerd epic](https://issues.redhat.com/browse/WINC-505) | ||
|
|
||
| ### Justification | ||
|
|
||
| *If we don't make containerd as default runtime, we will be left out with Mirantis supported dockershim | ||
| and will have to re-design WMCO to incorporate Mirantis dockershim. | ||
| *From the business point of view subscribing to Mirantis for dockershim support is not an ideal situation as Mirantis | ||
| is our key competitor. | ||
| *If we want to use CRI-O for Windows, the amount of time and engineering efforts involved making CRI-O to work for | ||
| Windows is huge. This doesn't do business justification due to the high cost and time. | ||
| *Containerd is widely adopted and supported by the open source community. | ||
| *Microsoft is a major contributor in Windows containerd development and moved to containerd as default runtime for their | ||
| kubernetes offerings. | ||
| *Most of the Windows-supported kubernetes orchestrators already moved to containerd. | ||
| *Containerd has become the default runtime in linux/windows nodes. | ||
| *Testing and adaptation will require less engineering effort compare to other solutions due to wider open source | ||
| community and enterprise support. | ||
|
|
||
| ### Design Details | ||
|
|
||
| we plan to use containerd 1.6.0 to integrate into WMCO. Containerd will be installed as a first service before | ||
| kubelet installation as kubelet must require containerd to be running. We already bundled containerd as part of | ||
| WMCO package. As we can use ctr commandline to test containerd, crictl commandline will not be installed. | ||
| Once containerd is installed, the rest of the service installation steps remain the same. Kubelet's dependency on Docker | ||
| runtime will be removed and replaced by containerd. A separate folder will be created under C:\k\ to store containerd | ||
| config and related files. In BYOH case, even after containerd becomes default runtime Docker will continue to run. | ||
| User can choose to uninstall Docker or use it for non-kubernetes related requirements. As docker installation is | ||
| handled by the user, it is up to the user to decide if Docker should continue to run or not. | ||
| When it comes to MachineSet, we can choose a VM image which doesn't have Docker. Containerd will be the only runtime. | ||
| Even if Docker gets installed at any time, it can co-exist with containerd. | ||
| steps to install containerd as service | ||
| *scp containerd/related executables into Windows VM | ||
| *copy the files in appropriate folder | ||
| *create containerd config file | ||
| *run below commands in PowerShell cmdlet | ||
| *run extra command for performance `Add-MpPreference -ExclusionProcess "$Env:containerd.exe"` | ||
| *register containerd as service | ||
| *start containerd as a Windows service | ||
|
|
||
| ## Network changes | ||
| Current CNI/IPAM will be used for containerd and no change in HNS-Network and HNS-Endpoint creation | ||
| steps. The config file which will be used by containerd will point to same CNI/IPAM executables. | ||
|
|
||
| ## WMCO feature flag | ||
| Containerd feature flag will be used to enable this feature. This will be enabled by default in | ||
| community operator for customers and developers to try it out. This will become the default for OpenShift | ||
| 4.11. At any given point in time, we have one runtime support and do not allow switching between runtime. | ||
|
|
||
| ## logging | ||
| Containerd can be started with parameters in which we can enable logging and specify the file path to | ||
| log warnings/errors. Log files will be stored at c:\var\log\containerd | ||
|
|
||
| ## upgrade | ||
| When a cluster is upgraded OLM will switch to using a new Red Hat operators | ||
| index. Because WMCO is named the same in both indexes, OLM will upgrade WMCO | ||
| from the previous version, up to the latest version available in the new | ||
| cluster. For further details please refer [WMCO-Upgrade](https://github.com/openshift/enhancements/blob/master/enhancements/windows-containers/windows-machine-config-operator-upgrades.md) | ||
|
|
||
| The procedure for an upgrade is as follows: | ||
| 1) MachineSet | ||
| * As part of upgrade Machine object gets deleted, resulting in the drain and deletion of the Windows node. | ||
| * New VM will be created. | ||
| * The upgraded WMCO instance will then be able to configure VM that will be created with containerd as | ||
| default runtime. | ||
| * Once the Windows node joins kubernetes cluster, user can deploy pods on the Windows node. | ||
| 2) BYOH ( Bring Your Own Host) | ||
| * upgrade process will uninstall all components that were installed by WMCO. | ||
| * As part of the de-configuring process Windows OS-specific configurations like HNS-Network that were created by WMCO | ||
| would be deleted. | ||
| * Once cleanup is done, reinstall workflow will install containerd along with kubelet, kube-proxy, CNI and | ||
| hybrid-overlay. | ||
| * Kuebelet will start using containerd as default runtime. | ||
| * Once the Windows node joins kubernetes cluster, user can deploy pods on the Windows node. | ||
|
|
||
| ### Risks and Mitigations | ||
|
|
||
| * If the cluster is upgraded, and the new version introduces an issue due to containerd | ||
| that should be addressed because docker runtime support won't be available in kubelet. | ||
| As we don't support downgrade, reverting to an older version is not possible. To overcome | ||
| any issues, we introduce this feature as part of the 5.0 community operator so that this feature | ||
| can be well tested before we make it default in WMCO 6.0 release. | ||
| * If we run into any issue in containerd, we work closely with containerd open source community and try | ||
| to resolve this as soon as possible. | ||
| * We are creating a new test job to test all e2e test cases in the community operator branch to capture any regression. | ||
| This will help us to identify any issues well before containerd becomes default runtime. | ||
| * containerd doesn't support image-pull-progress-deadline as of now. There is a PR | ||
| https://github.com/containerd/containerd/pull/6150 work in progress. Until this | ||
| gets merged, if Windows image pull takes more time than the default value, we might | ||
| run into to image pull timeout error. To overcome this issue, user should login to Windows VM and then pull the | ||
| image first with ctr commandline tool. Upon successful image download, user can create pods. | ||
| * Currently Windows_exporter has been used to collect metrics from Windows node. we do | ||
| have containerd support in https://github.com/prometheus-community/Windows_exporter/releases/tag/v0.16.0 | ||
| As of now there is no parity mismatch between Docker CRI and containerd. | ||
|
|
||
| ### Test Plan | ||
|
|
||
| * We will create a new test job (OKD) in the 4.10 community branch. We will pass WMCO feature flag in the test job | ||
| so that containerd will be enabled as runtime. All exiting e2e tests can be run on one or two platforms. | ||
| This will cover all existing test cases. We should make sure there is no regression due to containerd. | ||
| * Containerd is agnostic to the platform so testing in any platform should be fine. | ||
|
|
||
| ### Graduation Criteria | ||
|
|
||
| This enhancement will start with WMCO 5.0 community operator. Customers can test this feature with OpenShift 4.10 | ||
| This feature will be enabled by default from WMCO 6.0 (OpenShift 4.11) release onwards. | ||
|
|
||
| ### Upgrade / Downgrade Strategy | ||
|
|
||
| The upgrade is already discussed in the design section. Downgrades are [not supported](https://github.com/operator-framework/operator-lifecycle-manager/issues/1177) | ||
| by OLM. | ||
|
|
||
| ### Version Skew Strategy | ||
| We plan to maintain parity with the upstream [containerd](https://github.com/containerd/containerd/releases) | ||
| As part of the existing submodules update process, containerd will also be updated to a newer version. | ||
|
|
||
| ## Implementation History | ||
|
|
||
| v1: Initial Proposal | ||
|
|
||
| ## Alternatives | ||
|
|
||
| There are few alternatives, but they are either not cost-effective or depending on competitor's less modular | ||
| components. | ||
| * Implementing CRIO runtime for Windows involves huge engineering effort and there no community | ||
| support ( most community supporters already moved to containerd). | ||
| * There is an effort going on to continue to use dockershim and docker runtime. As kubelet is going to | ||
| remove dockershim specific code, we still have to come up with a design change to make it work from kubernetes 1.24 | ||
| onwards. |