From 98a31da88adbd7a4dc15d074259e60d47fdc472b Mon Sep 17 00:00:00 2001 From: Mark Rossetti Date: Tue, 31 May 2022 15:31:44 -0700 Subject: [PATCH 1/2] Updating some Windows docs pages including: - Fixinig heading levels for /docs/concepts/windows/intro.md - Misc updates to /docs/concepts/windows/intro.ms - Updates to /docs/concepts/configuration/windows-resource-management.md for accuracy Signed-off-by: Mark Rossetti --- .../windows-resource-management.md | 61 +++++++++---------- content/en/docs/concepts/windows/intro.md | 41 ++++++------- 2 files changed, 48 insertions(+), 54 deletions(-) diff --git a/content/en/docs/concepts/configuration/windows-resource-management.md b/content/en/docs/concepts/configuration/windows-resource-management.md index 6593caa5fb09c..658306326ff8b 100644 --- a/content/en/docs/concepts/configuration/windows-resource-management.md +++ b/content/en/docs/concepts/configuration/windows-resource-management.md @@ -32,52 +32,47 @@ host, and thus privileged containers are not available on Windows. Containers cannot assume an identity from the host because the Security Account Manager (SAM) is separate. -## Memory reservations {#resource-management-memory} +## Memory management {#resource-management-memory} Windows does not have an out-of-memory process killer as Linux does. Windows always treats all user-mode memory allocations as virtual, and pagefiles are mandatory. -Windows nodes do not overcommit memory for processes running in containers. The +Windows nodes do not overcommit memory for processes. The net effect is that Windows won't reach out of memory conditions the same way Linux does, and processes page to disk instead of being subject to out of memory (OOM) termination. If memory is over-provisioned and all physical memory is exhausted, then paging can slow down performance. -You can place bounds on memory use for workloads using the kubelet -parameters `--kubelet-reserve` and/or `--system-reserve`; these account -for memory usage on the node (outside of containers), and reduce -[NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable). -As you deploy workloads, set resource limits on containers. This also subtracts from -`NodeAllocatable` and prevents the scheduler from adding more pods once a node is full. - -{{< note >}} -When you set memory resource limits for Windows containers, you should either set a -limit and leave the memory request unspecified, or set the request equal to the limit. -{{< /note >}} - -On Windows, good practice to avoid over-provisioning is to configure the kubelet -with a system reserved memory of at least 2GiB to account for Windows, Kubernetes -and container runtime overheads. - -## CPU reservations {#resource-management-cpu} - -To account for CPU use by the operating system, the container runtime, and by -Kubernetes host processes such as the kubelet, you can (and should) reserve a -percentage of total CPU. You should determine this CPU reservation taking account of -to the number of CPU cores available on the node. To decide on the CPU percentage to -reserve, identify the maximum pod density for each node and monitor the CPU usage of -the system services running there, then choose a value that meets your workload needs. +## CPU management {#resource-management-cpu} -You can place bounds on CPU usage for workloads using the -kubelet parameters `--kubelet-reserve` and/or `--system-reserve` to -account for CPU usage on the node (outside of containers). -This reduces `NodeAllocatable`. -The cluster-wide scheduler then takes this reservation into account when determining -pod placement. +Windows can limit the amount of CPU time allocated for different processes but cannot +gaurentee a miniumum amount of CPU time. -On Windows, the kubelet supports a command-line flag to set the priority of the +On Windows, the kubelet supports a command-line flag to set the +[scheduling priority](https://docs.microsoft.com/windows/win32/procthread/scheduling-priorities) of the kubelet process: `--windows-priorityclass`. This flag allows the kubelet process to get more CPU time slices when compared to other processes running on the Windows host. More information on the allowable values and their meaning is available at [Windows Priority Classes](https://docs.microsoft.com/en-us/windows/win32/procthread/scheduling-priorities#priority-class). To ensure that running Pods do not starve the kubelet of CPU cycles, set this flag to `ABOVE_NORMAL_PRIORITY_CLASS` or above. + +## Resource Reservation {$resource-reservation} + +To account for memory and CPU used by the operating system, the container runtime, and by +Kubernetes host processes such as the kubelet, you can (and should) reserve +memory and CPU resrouces with the `--kube-reserved` and/or `--system-reserved` kubelet parameters. +On Windows these values are only used to calculate the node's [allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) resources. + +{{< caution >}} +As you deploy workloads, set resource memory and CPU limits on containers. +This also subtracts from `NodeAllocatable` and helps the cluster-wide scheduler in determining which pods to place on which nodes. + +Scheduling pods without limits may over-provision the Windows nodes and in extreme +cases can cause the nodes to become unhealthy. +{{< /caution >}} + +On Windows, good practice is to reserve at least 2GiB of memory. + +To determine how much CPU to reservce, +identify the maximum pod density for each node and monitor the CPU usage of +the system services running there, then choose a value that meets your workload needs. diff --git a/content/en/docs/concepts/windows/intro.md b/content/en/docs/concepts/windows/intro.md index a737d54e3b1c0..f118ef199a587 100644 --- a/content/en/docs/concepts/windows/intro.md +++ b/content/en/docs/concepts/windows/intro.md @@ -50,7 +50,8 @@ Some node features are only available if you use a specific including: * HugePages: not supported for Windows containers -* Privileged containers: not supported for Windows containers +* Privileged containers: not supported for Windows containers. + [HostProcess Containers](/docs/tasks/configure-pod-container/create-hostprocess-pod.md) offer similar functionality. * TerminationGracePeriod: requires containerD Not all features of shared namespaces are supported. See [API compatibility](#api) @@ -78,7 +79,7 @@ section refers to several key workload abstractions and how they map to Windows. * Single or multiple containers per Pod with process isolation and volume sharing * Pod `status` fields - * Readiness and Liveness probes + * Readiness, Liveness, and startup probes * postStart & preStop container lifecycle hooks * ConfigMap, Secrets: as environment variables or volumes * `emptyDir` volumes @@ -136,7 +137,7 @@ section refers to several key workload abstractions and how they map to Windows. Pods, workload resources, and Services are critical elements to managing Windows workloads on Kubernetes. However, on their own they are not enough to enable the proper lifecycle management of Windows workloads in a dynamic cloud native -environment. Kubernetes also supports: +environment. * `kubectl exec` * Pod and container metrics @@ -150,14 +151,13 @@ Some kubelet command line options behave differently on Windows, as described be * The `--windows-priorityclass` lets you set the scheduling priority of the kubelet process (see [CPU resource management](/docs/concepts/configuration/windows-resource-management/#resource-management-cpu)) -* The `--kubelet-reserve`, `--system-reserve` , and `--eviction-hard` flags update +* The `--kube-reserved`, `--system-reserved` , and `--eviction-hard` flags update [NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) * Eviction by using `--enforce-node-allocable` is not implemented * Eviction by using `--eviction-hard` and `--eviction-soft` are not implemented -* A kubelet running on a Windows node does not have memory - restrictions. `--kubelet-reserve` and `--system-reserve` do not set limits on - kubelet or processes running on the host. This means kubelet or a process on the host - could cause memory resource starvation outside the node-allocatable and scheduler. +* When running on a Windows node the kubelet does not have memory or CPU + restrictions. `--kube-reserved` and `--system-reserved` only subtract from `NodeAllocatable` and do not gaurentee resources. + See [Resource Management for Windows nodes](/docs/concepts/configuration/windows-resource-management/#resource-reservation) for more information. * The `MemoryPressure` Condition is not implemented * The kubelet does not take OOM eviction actions @@ -192,7 +192,7 @@ Container exit codes follow the same convention where 0 is success, and nonzero The specific error codes may differ across Windows and Linux. However, exit codes passed from the Kubernetes components (kubelet, kube-proxy) are unchanged. -##### Field compatibility for container specifications {#compatibility-v1-pod-spec-containers} +#### Field compatibility for container specifications {#compatibility-v1-pod-spec-containers} The following list documents differences between how Pod container specifications work between Windows and Linux: @@ -232,7 +232,7 @@ work between Windows and Linux: default value is `/dev/termination-log`, which does work because it does not exist on Windows by default. -##### Field compatibility for Pod specifications {#compatibility-v1-pod} +#### Field compatibility for Pod specifications {#compatibility-v1-pod} The following list documents differences between how Pod specifications work between Windows and Linux: @@ -260,17 +260,18 @@ The following list documents differences between how Pod specifications work bet * You cannot enable `mountPropagation` for volume mounts as this is not supported on Windows. -##### Field compatibility for Pod security context {#compatibility-v1-pod-spec-containers-securitycontext} +#### Field compatibility for Pod security context {#compatibility-v1-pod-spec-containers-securitycontext} None of the Pod [`securityContext`](/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context) fields work on Windows. -### Node problem detector +## Node problem detector The node problem detector (see [Monitor Node Health](/docs/tasks/debug/debug-cluster/monitor-node-health/)) -is not compatible with Windows. +has prelimiary support for Windows. +For more information visit the the project's [GitHub page](https://github.com/kubernetes/node-problem-detector#windows) -### Pause container +## Pause container In a Kubernetes Pod, an infrastructure or “pause” container is first created to host the container. In Linux, the cgroups and namespaces that make up a pod @@ -293,7 +294,7 @@ The Kubernetes project recommends using the Microsoft maintained image if you ar deploying to a production or production-like environment that requires signed binaries. -### Container runtimes {#container-runtime} +## Container runtimes {#container-runtime} You need to install a {{< glossary_tooltip text="container runtime" term_id="container-runtime" >}} @@ -303,7 +304,7 @@ The following container runtimes work with Windows: {{% thirdparty-content %}} -#### cri-containerd +### ContainerD {{< feature-state for_k8s_version="v1.20" state="stable" >}} @@ -318,7 +319,7 @@ when using GMSA with containerd to access Windows network shares, which requires kernel patch. {{< /note >}} -#### Mirantis Container Runtime {#mcr} +### Mirantis Container Runtime {#mcr} [Mirantis Container Runtime](https://docs.mirantis.com/mcr/20.10/overview.html) (MCR) is available as a container runtime for all Windows Server 2019 and later versions. @@ -364,9 +365,7 @@ reported previously and comment with your experience on the issue and add additi logs. SIG-Windows Slack is also a great avenue to get some initial support and troubleshooting ideas prior to creating a ticket. -## {{% heading "whatsnext" %}} - -### Deployment tools +## Deployment tools The kubeadm tool helps you to deploy a Kubernetes cluster, providing the control plane to manage the cluster it, and nodes to run your workloads. @@ -375,7 +374,7 @@ explains how to deploy Windows nodes to your cluster using kubeadm. The Kubernetes [cluster API](https://cluster-api.sigs.k8s.io/) project also provides means to automate deployment of Windows nodes. -### Windows distribution channels +## Windows distribution channels For a detailed explanation of Windows distribution channels see the [Microsoft documentation](https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19). From db6814f067e3795a3e0a17882544c351faa3f71c Mon Sep 17 00:00:00 2001 From: Mark Rossetti Date: Thu, 2 Jun 2022 15:17:54 -0700 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: Qiming Teng Co-authored-by: Tim Bannister Co-authored-by: divya-mohan0209 --- .../windows-resource-management.md | 13 ++++++----- content/en/docs/concepts/windows/intro.md | 22 +++++++++++-------- 2 files changed, 20 insertions(+), 15 deletions(-) diff --git a/content/en/docs/concepts/configuration/windows-resource-management.md b/content/en/docs/concepts/configuration/windows-resource-management.md index 658306326ff8b..955fea194a4b1 100644 --- a/content/en/docs/concepts/configuration/windows-resource-management.md +++ b/content/en/docs/concepts/configuration/windows-resource-management.md @@ -46,7 +46,7 @@ then paging can slow down performance. ## CPU management {#resource-management-cpu} Windows can limit the amount of CPU time allocated for different processes but cannot -gaurentee a miniumum amount of CPU time. +guarantee a minimum amount of CPU time. On Windows, the kubelet supports a command-line flag to set the [scheduling priority](https://docs.microsoft.com/windows/win32/procthread/scheduling-priorities) of the @@ -56,12 +56,13 @@ More information on the allowable values and their meaning is available at [Windows Priority Classes](https://docs.microsoft.com/en-us/windows/win32/procthread/scheduling-priorities#priority-class). To ensure that running Pods do not starve the kubelet of CPU cycles, set this flag to `ABOVE_NORMAL_PRIORITY_CLASS` or above. -## Resource Reservation {$resource-reservation} +## Resource reservation {#resource-reservation} To account for memory and CPU used by the operating system, the container runtime, and by Kubernetes host processes such as the kubelet, you can (and should) reserve -memory and CPU resrouces with the `--kube-reserved` and/or `--system-reserved` kubelet parameters. -On Windows these values are only used to calculate the node's [allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) resources. +memory and CPU resources with the `--kube-reserved` and/or `--system-reserved` kubelet flags. +On Windows these values are only used to calculate the node's +[allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) resources. {{< caution >}} As you deploy workloads, set resource memory and CPU limits on containers. @@ -71,8 +72,8 @@ Scheduling pods without limits may over-provision the Windows nodes and in extre cases can cause the nodes to become unhealthy. {{< /caution >}} -On Windows, good practice is to reserve at least 2GiB of memory. +On Windows, a good practice is to reserve at least 2GiB of memory. -To determine how much CPU to reservce, +To determine how much CPU to reserve, identify the maximum pod density for each node and monitor the CPU usage of the system services running there, then choose a value that meets your workload needs. diff --git a/content/en/docs/concepts/windows/intro.md b/content/en/docs/concepts/windows/intro.md index f118ef199a587..91e9412759fe5 100644 --- a/content/en/docs/concepts/windows/intro.md +++ b/content/en/docs/concepts/windows/intro.md @@ -51,7 +51,7 @@ including: * HugePages: not supported for Windows containers * Privileged containers: not supported for Windows containers. - [HostProcess Containers](/docs/tasks/configure-pod-container/create-hostprocess-pod.md) offer similar functionality. + [HostProcess Containers](/docs/tasks/configure-pod-container/create-hostprocess-pod/) offer similar functionality. * TerminationGracePeriod: requires containerD Not all features of shared namespaces are supported. See [API compatibility](#api) @@ -79,7 +79,7 @@ section refers to several key workload abstractions and how they map to Windows. * Single or multiple containers per Pod with process isolation and volume sharing * Pod `status` fields - * Readiness, Liveness, and startup probes + * Readiness, liveness, and startup probes * postStart & preStop container lifecycle hooks * ConfigMap, Secrets: as environment variables or volumes * `emptyDir` volumes @@ -156,8 +156,10 @@ Some kubelet command line options behave differently on Windows, as described be * Eviction by using `--enforce-node-allocable` is not implemented * Eviction by using `--eviction-hard` and `--eviction-soft` are not implemented * When running on a Windows node the kubelet does not have memory or CPU - restrictions. `--kube-reserved` and `--system-reserved` only subtract from `NodeAllocatable` and do not gaurentee resources. - See [Resource Management for Windows nodes](/docs/concepts/configuration/windows-resource-management/#resource-reservation) for more information. + restrictions. `--kube-reserved` and `--system-reserved` only subtract from `NodeAllocatable` + and do not guarantee resource provided for workloads. + See [Resource Management for Windows nodes](/docs/concepts/configuration/windows-resource-management/#resource-reservation) + for more information. * The `MemoryPressure` Condition is not implemented * The kubelet does not take OOM eviction actions @@ -268,8 +270,8 @@ None of the Pod [`securityContext`](/docs/reference/kubernetes-api/workload-reso The node problem detector (see [Monitor Node Health](/docs/tasks/debug/debug-cluster/monitor-node-health/)) -has prelimiary support for Windows. -For more information visit the the project's [GitHub page](https://github.com/kubernetes/node-problem-detector#windows) +has preliminary support for Windows. +For more information, visit the project's [GitHub page](https://github.com/kubernetes/node-problem-detector#windows). ## Pause container @@ -321,7 +323,8 @@ kernel patch. ### Mirantis Container Runtime {#mcr} -[Mirantis Container Runtime](https://docs.mirantis.com/mcr/20.10/overview.html) (MCR) is available as a container runtime for all Windows Server 2019 and later versions. +[Mirantis Container Runtime](https://docs.mirantis.com/mcr/20.10/overview.html) (MCR) +is available as a container runtime for all Windows Server 2019 and later versions. See [Install MCR on Windows Servers](https://docs.mirantis.com/mcr/20.10/install/mcr-windows.html) for more information. @@ -362,7 +365,7 @@ If you have what looks like a bug, or you would like to make a feature request, please follow the [SIG Windows contributing guide](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#reporting-issues-and-feature-requests) to create a new issue. You should first search the list of issues in case it was reported previously and comment with your experience on the issue and add additional -logs. SIG-Windows Slack is also a great avenue to get some initial support and +logs. SIG Windows channel on the Kubernetes Slack is also a great avenue to get some initial support and troubleshooting ideas prior to creating a ticket. ## Deployment tools @@ -376,7 +379,8 @@ The Kubernetes [cluster API](https://cluster-api.sigs.k8s.io/) project also prov ## Windows distribution channels -For a detailed explanation of Windows distribution channels see the [Microsoft documentation](https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19). +For a detailed explanation of Windows distribution channels see the +[Microsoft documentation](https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19). Information on the different Windows Server servicing channels including their support models can be found at