Merge pull request #26153 from kubernetes/dev-1.21

Official 1.21 Release Docs

config.toml

@@ -138,10 +138,10 @@ time_format_default = "January 02, 2006 at 3:04 PM PST"
 description = "Production-Grade Container Orchestration"
 showedit = true
 
-latest = "v1.20"
+latest = "v1.21"
 
-fullversion = "v1.20.0"
-version = "v1.20"
+fullversion = "v1.21.0"
+version = "v1.21"
 githubbranch = "master"
 docsbranch = "master"
 deprecated = false
@@ -178,40 +178,40 @@ js = [
 ]
 
 [[params.versions]]
-fullversion = "v1.20.0"
-version = "v1.20"
-githubbranch = "v1.20.0"
+fullversion = "v1.21.0"
+version = "v1.21"
+githubbranch = "v1.21.0"
 docsbranch = "master"
 url = "https://kubernetes.io"
 
 [[params.versions]]
-fullversion = "v1.19.4"
+fullversion = "v1.20.5"
+version = "v1.20"
+githubbranch = "v1.20.5"
+docsbranch = "release-1.20"
+url = "https://v1-20.kubernetes.io"
+
+[[params.versions]]
+fullversion = "v1.19.9"
 version = "v1.19"
-githubbranch = "v1.19.4"
+githubbranch = "v1.19.9"
 docsbranch = "release-1.19"
 url = "https://v1-19.docs.kubernetes.io"
 
 [[params.versions]]
-fullversion = "v1.18.12"
+fullversion = "v1.18.17"
 version = "v1.18"
-githubbranch = "v1.18.12"
+githubbranch = "v1.18.17"
 docsbranch = "release-1.18"
 url = "https://v1-18.docs.kubernetes.io"
 
 [[params.versions]]
-fullversion = "v1.17.14"
+fullversion = "v1.17.17"
 version = "v1.17"
-githubbranch = "v1.17.14"
+githubbranch = "v1.17.17"
 docsbranch = "release-1.17"
 url = "https://v1-17.docs.kubernetes.io"
 
-[[params.versions]]
-fullversion = "v1.16.15"
-version = "v1.16"
-githubbranch = "v1.16.15"
-docsbranch = "release-1.16"
-url = "https://v1-16.docs.kubernetes.io"
-
 
 # User interface configuration
 [params.ui]

content/en/docs/concepts/architecture/cloud-controller.md

@@ -206,6 +206,8 @@ rules:
 [Cloud Controller Manager Administration](/docs/tasks/administer-cluster/running-cloud-controller/#cloud-controller-manager)
 has instructions on running and managing the cloud controller manager.
 
+To upgrade a HA control plane to use the cloud controller manager, see [Migrate Replicated Control Plane To Use Cloud Controller Manager](/docs/tasks/administer-cluster/controller-manager-leader-migration/).
+
 Want to know how to implement your own cloud controller manager, or extend an existing project?
 
 The cloud controller manager uses Go interfaces to allow implementations from any cloud to be plugged in. Specifically, it uses the `CloudProvider` interface defined in [`cloud.go`](https://github.com/kubernetes/cloud-provider/blob/release-1.17/cloud.go#L42-L62) from [kubernetes/cloud-provider](https://github.com/kubernetes/cloud-provider).

content/en/docs/concepts/architecture/nodes.md

@@ -346,26 +346,43 @@ the kubelet can use topology hints when making resource assignment decisions.
 See [Control Topology Management Policies on a Node](/docs/tasks/administer-cluster/topology-manager/)
 for more information.
 
-## Graceful Node Shutdown {#graceful-node-shutdown}
+## Graceful node shutdown {#graceful-node-shutdown}
 
-{{< feature-state state="alpha" for_k8s_version="v1.20" >}}
+{{< feature-state state="beta" for_k8s_version="v1.21" >}}
+
+The kubelet attempts to detect node system shutdown and terminates pods running on the node.
 
-If you have enabled the `GracefulNodeShutdown` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/), then the kubelet attempts to detect the node system shutdown and terminates pods running on the node.
 Kubelet ensures that pods follow the normal [pod termination process](/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination) during the node shutdown.
 
-When the `GracefulNodeShutdown` feature gate is enabled, kubelet uses [systemd inhibitor locks](https://www.freedesktop.org/wiki/Software/systemd/inhibit/) to delay the node shutdown with a given duration. During a shutdown, kubelet terminates pods in two phases:
+The Graceful node shutdown feature depends on systemd since it takes advantage of
+[systemd inhibitor locks](https://www.freedesktop.org/wiki/Software/systemd/inhibit/) to
+delay the node shutdown with a given duration.
+
+Graceful node shutdown is controlled with the `GracefulNodeShutdown`
+[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) which is
+enabled by default in 1.21.
+
+Note that by default, both configuration options described below,
+`ShutdownGracePeriod` and `ShutdownGracePeriodCriticalPods` are set to zero,
+thus not activating Graceful node shutdown functionality.
+To activate the feature, the two kubelet config settings should be configured appropriately and set to non-zero values.
+
+During a graceful shutdown, kubelet terminates pods in two phases:
 
 1. Terminate regular pods running on the node.
 2. Terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) running on the node.
 
-Graceful Node Shutdown feature is configured with two [`KubeletConfiguration`](/docs/tasks/administer-cluster/kubelet-config-file/) options:
+Graceful node shutdown feature is configured with two [`KubeletConfiguration`](/docs/tasks/administer-cluster/kubelet-config-file/) options:
 * `ShutdownGracePeriod`:
   * Specifies the total duration that the node should delay the shutdown by. This is the total grace period for pod termination for both regular and [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical).
 * `ShutdownGracePeriodCriticalPods`:
-  * Specifies the duration used to terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) during a node shutdown. This should be less than `ShutdownGracePeriod`.
-
-For example, if `ShutdownGracePeriod=30s`, and `ShutdownGracePeriodCriticalPods=10s`, kubelet will delay the node shutdown by 30 seconds. During the shutdown, the first 20 (30-10) seconds would be reserved for gracefully terminating normal pods, and the last 10 seconds would be reserved for terminating [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical).
+  * Specifies the duration used to terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) during a node shutdown. This value should be less than `ShutdownGracePeriod`.
 
+For example, if `ShutdownGracePeriod=30s`, and
+`ShutdownGracePeriodCriticalPods=10s`, kubelet will delay the node shutdown by
+30 seconds. During the shutdown, the first 20 (30-10) seconds would be reserved
+for gracefully terminating normal pods, and the last 10 seconds would be
+reserved for terminating [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical).
 
 ## {{% heading "whatsnext" %}}
 

content/en/docs/concepts/cluster-administration/logging.md

@@ -83,12 +83,15 @@ As an example, you can find detailed information about how `kube-up.sh` sets
 up logging for COS image on GCP in the corresponding
 [`configure-helper` script](https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/cluster/gce/gci/configure-helper.sh).
 
+When using a **CRI container runtime**, the kubelet is responsible for rotating the logs and managing the logging directory structure. The kubelet
+sends this information to the CRI container runtime and the runtime writes the container logs to the given location. The two kubelet flags `container-log-max-size` and `container-log-max-files` can be used to configure the maximum size for each log file and the maximum number of files allowed for each container respectively.
+
 When you run [`kubectl logs`](/docs/reference/generated/kubectl/kubectl-commands#logs) as in
 the basic logging example, the kubelet on the node handles the request and
 reads directly from the log file. The kubelet returns the content of the log file.
 
 {{< note >}}
-If an external system has performed the rotation,
+If an external system has performed the rotation or a CRI container runtime is used,
 only the contents of the latest log file will be available through
 `kubectl logs`. For example, if there's a 10MB file, `logrotate` performs
 the rotation and there are two files: one file that is 10MB in size and a second file that is empty.

content/en/docs/concepts/cluster-administration/system-metrics.md

@@ -134,7 +134,7 @@ cloudprovider_gce_api_request_duration_seconds { request = "list_disk"}
 
 ### kube-scheduler metrics
 
-{{< feature-state for_k8s_version="v1.20" state="alpha" >}}
+{{< feature-state for_k8s_version="v1.21" state="beta" >}}
 
 The scheduler exposes optional metrics that reports the requested resources and the desired limits of all running pods. These metrics can be used to build capacity planning dashboards, assess current or historical scheduling limits, quickly identify workloads that cannot schedule due to lack of resources, and compare actual usage to the pod's request.
 
@@ -152,6 +152,24 @@ Once a pod reaches completion (has a `restartPolicy` of `Never` or `OnFailure` a
 The metrics are exposed at the HTTP endpoint `/metrics/resources` and require the same authorization as the `/metrics`
 endpoint on the scheduler. You must use the `--show-hidden-metrics-for-version=1.20` flag to expose these alpha stability metrics.
 
+## Disabling metrics
+
+You can explicitly turn off metrics via command line flag `--disabled-metrics`. This may be desired if, for example, a metric is causing a performance problem. The input is a list of disabled metrics (i.e. `--disabled-metrics=metric1,metric2`).
+
+## Metric cardinality enforcement
+
+Metrics with unbounded dimensions could cause memory issues in the components they instrument. To limit resource use, you can use the `--allow-label-value` command line option to dynamically configure an allow-list of label values for a metric.
+
+In alpha stage, the flag can only take in a series of mappings as metric label allow-list.
+Each mapping is of the format `<metric_name>,<label_name>=<allowed_labels>` where 
+`<allowed_labels>` is a comma-separated list of acceptable label names.
+
+The overall format looks like:
+`--allow-label-value <metric_name>,<label_name>='<allow_value1>, <allow_value2>...', <metric_name2>,<label_name>='<allow_value1>, <allow_value2>...', ...`.
+
+Here is an example:
+`--allow-label-value number_count_metric,odd_number='1,3,5', number_count_metric,even_number='2,4,6', date_gauge_metric,weekend='Saturday,Sunday'`
+
 
 ## {{% heading "whatsnext" %}}
 

content/en/docs/concepts/configuration/configmap.md

@@ -236,9 +236,9 @@ ConfigMaps consumed as environment variables are not updated automatically and r
 
 ## Immutable ConfigMaps {#configmap-immutable}
 
-{{< feature-state for_k8s_version="v1.19" state="beta" >}}
+{{< feature-state for_k8s_version="v1.21" state="stable" >}}
 
-The Kubernetes beta feature _Immutable Secrets and ConfigMaps_ provides an option to set
+The Kubernetes feature _Immutable Secrets and ConfigMaps_ provides an option to set
 individual Secrets and ConfigMaps as immutable. For clusters that extensively use ConfigMaps
 (at least tens of thousands of unique ConfigMap to Pod mounts), preventing changes to their
 data has the following advantages:

content/en/docs/concepts/configuration/secret.md

@@ -749,9 +749,9 @@ There are third party solutions for triggering restarts when secrets change.
 
 ## Immutable Secrets {#secret-immutable}
 
-{{< feature-state for_k8s_version="v1.19" state="beta" >}}
+{{< feature-state for_k8s_version="v1.21" state="stable" >}}
 
-The Kubernetes beta feature _Immutable Secrets and ConfigMaps_ provides an option to set
+The Kubernetes feature _Immutable Secrets and ConfigMaps_ provides an option to set
 individual Secrets and ConfigMaps as immutable. For clusters that extensively use Secrets
 (at least tens of thousands of unique Secret to Pod mounts), preventing changes to their
 data has the following advantages:

content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md

@@ -193,9 +193,69 @@ for these devices:
 // node resources consumed by pods and containers on the node
 service PodResourcesLister {
     rpc List(ListPodResourcesRequest) returns (ListPodResourcesResponse) {}
+    rpc GetAllocatableResources(AllocatableResourcesRequest) returns (AllocatableResourcesResponse) {}
 }
 ```
 
+The `List` endpoint provides information on resources of running pods, with details such as the
+id of exclusively allocated CPUs, device id as it was reported by device plugins and id of
+the NUMA node where these devices are allocated.
+
+```gRPC
+// ListPodResourcesResponse is the response returned by List function
+message ListPodResourcesResponse {
+    repeated PodResources pod_resources = 1;
+}
+
+// PodResources contains information about the node resources assigned to a pod
+message PodResources {
+    string name = 1;
+    string namespace = 2;
+    repeated ContainerResources containers = 3;
+}
+
+// ContainerResources contains information about the resources assigned to a container
+message ContainerResources {
+    string name = 1;
+    repeated ContainerDevices devices = 2;
+    repeated int64 cpu_ids = 3;
+}
+
+// Topology describes hardware topology of the resource
+message TopologyInfo {
+        repeated NUMANode nodes = 1;
+}
+
+// NUMA representation of NUMA node
+message NUMANode {
+        int64 ID = 1;
+}
+
+// ContainerDevices contains information about the devices assigned to a container
+message ContainerDevices {
+    string resource_name = 1;
+    repeated string device_ids = 2;
+    TopologyInfo topology = 3;
+}
+```
+
+GetAllocatableResources provides information on resources initially available on the worker node.
+It provides more information than kubelet exports to APIServer.
+
+```gRPC
+// AllocatableResourcesResponses contains informations about all the devices known by the kubelet
+message AllocatableResourcesResponse {
+    repeated ContainerDevices devices = 1;
+    repeated int64 cpu_ids = 2;
+}
+
+```
+
+`ContainerDevices` do expose the topology information declaring to which NUMA cells the device is affine.
+The NUMA cells are identified using a opaque integer ID, which value is consistent to what device
+plugins report [when they register themselves to the kubelet](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-integration-with-the-topology-manager).
+
+
 The gRPC service is served over a unix socket at `/var/lib/kubelet/pod-resources/kubelet.sock`.
 Monitoring agents for device plugin resources can be deployed as a daemon, or as a DaemonSet.
 The canonical directory `/var/lib/kubelet/pod-resources` requires privileged access, so monitoring
@@ -204,7 +264,7 @@ DaemonSet, `/var/lib/kubelet/pod-resources` must be mounted as a
 {{< glossary_tooltip term_id="volume" >}} in the device monitoring agent's
 [PodSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podspec-v1-core).
 
-Support for the "PodResources service" requires `KubeletPodResources` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to be enabled.
+Support for the `PodResourcesLister service` requires `KubeletPodResources` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to be enabled.
 It is enabled by default starting with Kubernetes 1.15 and is v1 since Kubernetes 1.20.
 
 ## Device Plugin integration with the Topology Manager
@@ -256,5 +316,3 @@ Here are some examples of device plugin implementations:
 * Learn about [advertising extended resources](/docs/tasks/administer-cluster/extended-resource-node/) on a node
 * Read about using [hardware acceleration for TLS ingress](https://kubernetes.io/blog/2019/04/24/hardware-accelerated-ssl/tls-termination-in-ingress-controllers-using-kubernetes-device-plugins-and-runtimeclass/) with Kubernetes
 * Learn about the [Topology Manager](/docs/tasks/administer-cluster/topology-manager/)
-
-

content/en/docs/concepts/overview/working-with-objects/namespaces.md

@@ -30,7 +30,7 @@ Namespaces are a way to divide cluster resources between multiple users (via [re
 
 It is not necessary to use multiple namespaces to separate slightly different
 resources, such as different versions of the same software: use
-[labels](/docs/concepts/overview/working-with-objects/labels) to distinguish
+{{< glossary_tooltip text="labels" term_id="label" >}} to distinguish
 resources within the same namespace.
 
 ## Working with Namespaces
@@ -114,6 +114,16 @@ kubectl api-resources --namespaced=true
 kubectl api-resources --namespaced=false
 ```
 
+## Automatic labelling
+
+{{< feature-state state="beta" for_k8s_version="1.21" >}}
+
+The Kubernetes control plane sets an immutable {{< glossary_tooltip text="label" term_id="label" >}}
+`kubernetes.io/metadata.name` on all namespaces, provided that the `NamespaceDefaultLabelName`
+[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled.
+The value of the label is the namespace name.
+
+
 ## {{% heading "whatsnext" %}}
 
 * Learn more about [creating a new namespace](/docs/tasks/administer-cluster/namespaces/#creating-a-new-namespace).

content/en/docs/concepts/policy/node-resource-managers.md

@@ -0,0 +1,22 @@
+---
+reviewers:
+- derekwaynecarr
+- klueska
+title: Node Resource Managers 
+content_type: concept
+weight: 50
+---
+
+<!-- overview -->
+
+In order to support latency-critical and high-throughput workloads, Kubernetes offers a suite of Resource Managers. The managers aim to co-ordinate and optimise node's resources alignment for pods configured with a specific requirement for CPUs, devices, and memory (hugepages) resources. 
+
+<!-- body -->
+
+The main manager, the Topology Manager, is a Kubelet component that co-ordinates the overall resource management process through its [policy](/docs/tasks/administer-cluster/topology-manager/).
+
+The configuration of individual managers is elaborated in dedicated documents:
+
+- [CPU Manager Policies](/docs/tasks/administer-cluster/cpu-management-policies/)
+- [Device Manager](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-integration-with-the-topology-manager)
+- [Memory Manger Policies](/docs/tasks/administer-cluster/memory-manager/)

content/en/docs/concepts/policy/pod-security-policy.md

@@ -9,7 +9,9 @@ weight: 30
 
 <!-- overview -->
 
-{{< feature-state state="beta" >}}
+{{< feature-state for_k8s_version="v1.21" state="deprecated" >}}
+
+PodSecurityPolicy is deprecated as of Kubernetes v1.21, and will be removed in v1.25.
 
 Pod Security Policies enable fine-grained authorization of pod creation and
 updates.