From f59c9f7473c49c1c0a7ecbe80da663e8316bd412 Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Wed, 14 Jun 2023 12:31:58 +0800 Subject: [PATCH] Tweak line wrappings in configure-aggregation-layer.md Signed-off-by: xin.li --- .../configure-aggregation-layer.md | 185 +++++++++++++----- 1 file changed, 139 insertions(+), 46 deletions(-) diff --git a/content/en/docs/tasks/extend-kubernetes/configure-aggregation-layer.md b/content/en/docs/tasks/extend-kubernetes/configure-aggregation-layer.md index 6c07d5abcc3c0..89c07890af90f 100644 --- a/content/en/docs/tasks/extend-kubernetes/configure-aggregation-layer.md +++ b/content/en/docs/tasks/extend-kubernetes/configure-aggregation-layer.md @@ -10,31 +10,44 @@ weight: 10 -Configuring the [aggregation layer](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) allows the Kubernetes apiserver to be extended with additional APIs, which are not part of the core Kubernetes APIs. +Configuring the [aggregation layer](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) +allows the Kubernetes apiserver to be extended with additional APIs, which are not +part of the core Kubernetes APIs. ## {{% heading "prerequisites" %}} {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} {{< note >}} -There are a few setup requirements for getting the aggregation layer working in your environment to support mutual TLS auth between the proxy and extension apiservers. Kubernetes and the kube-apiserver have multiple CAs, so make sure that the proxy is signed by the aggregation layer CA and not by something else, like the Kubernetes general CA. +There are a few setup requirements for getting the aggregation layer working in +your environment to support mutual TLS auth between the proxy and extension apiservers. +Kubernetes and the kube-apiserver have multiple CAs, so make sure that the proxy is +signed by the aggregation layer CA and not by something else, like the Kubernetes general CA. {{< /note >}} {{< caution >}} -Reusing the same CA for different client types can negatively impact the cluster's ability to function. For more information, see [CA Reusage and Conflicts](#ca-reusage-and-conflicts). +Reusing the same CA for different client types can negatively impact the cluster's +ability to function. For more information, see [CA Reusage and Conflicts](#ca-reusage-and-conflicts). {{< /caution >}} ## Authentication Flow -Unlike Custom Resource Definitions (CRDs), the Aggregation API involves another server - your Extension apiserver - in addition to the standard Kubernetes apiserver. The Kubernetes apiserver will need to communicate with your extension apiserver, and your extension apiserver will need to communicate with the Kubernetes apiserver. In order for this communication to be secured, the Kubernetes apiserver uses x509 certificates to authenticate itself to the extension apiserver. +Unlike Custom Resource Definitions (CRDs), the Aggregation API involves +another server - your Extension apiserver - in addition to the standard Kubernetes apiserver. +The Kubernetes apiserver will need to communicate with your extension apiserver, +and your extension apiserver will need to communicate with the Kubernetes apiserver. +In order for this communication to be secured, the Kubernetes apiserver uses x509 +certificates to authenticate itself to the extension apiserver. -This section describes how the authentication and authorization flows work, and how to configure them. +This section describes how the authentication and authorization flows work, +and how to configure them. The high-level flow is as follows: -1. Kubernetes apiserver: authenticate the requesting user and authorize their rights to the requested API path. +1. Kubernetes apiserver: authenticate the requesting user and authorize their + rights to the requested API path. 2. Kubernetes apiserver: proxy the request to the extension apiserver 3. Extension apiserver: authenticate the request from the Kubernetes apiserver 4. Extension apiserver: authorize the request from the original user @@ -63,7 +76,8 @@ note: kube-apiserver / aggregator -> kube-apiserver / aggregator: authentication note: -2. The Kube API server authenticates the incoming request using any configured authentication methods (e.g. OIDC or client certs) +2. The Kube API server authenticates the incoming request using any configured + authentication methods (e.g. OIDC or client certs) kube-apiserver / aggregator -> kube-apiserver / aggregator: authorization @@ -73,8 +87,10 @@ note: kube-apiserver / aggregator -> aggregated apiserver: note: -4. The aggregator opens a connection to the aggregated API server using `--proxy-client-cert-file`/`--proxy-client-key-file` client certificate/key to secure the channel -5. The aggregator sends the user info from step 1 to the aggregated API server as http headers, as defined by the following flags: +4. The aggregator opens a connection to the aggregated API server using + `--proxy-client-cert-file`/`--proxy-client-key-file` client certificate/key to secure the channel +5. The aggregator sends the user info from step 1 to the aggregated API server as + http headers, as defined by the following flags: * `--requestheader-username-headers` * `--requestheader-group-headers` * `--requestheader-extra-headers-prefix` @@ -86,27 +102,41 @@ note: * verifies the request has a recognized auth proxy client certificate * pulls user info from the incoming request's http headers -By default, it pulls the configuration information for this from a configmap in the kube-system namespace that is published by the kube-apiserver, containing the info from the `--requestheader-...` flags provided to the kube-apiserver (CA bundle to use, auth proxy client certificate names to allow, http header names to use, etc) +By default, it pulls the configuration information for this from a configmap +in the kube-system namespace that is published by the kube-apiserver, +containing the info from the `--requestheader-...` flags provided to the +kube-apiserver (CA bundle to use, auth proxy client certificate names to allow, +http header names to use, etc) aggregated apiserver -> kube-apiserver / aggregator: authorization note: -7. The aggregated apiserver authorizes the incoming request by making a SubjectAccessReview call to the kube-apiserver +7. The aggregated apiserver authorizes the incoming request by making a + SubjectAccessReview call to the kube-apiserver aggregated apiserver -> aggregated apiserver: admission note: -8. For mutating requests, the aggregated apiserver runs admission checks. by default, the namespace lifecycle admission plugin ensures namespaced resources are created in a namespace that exists in the kube-apiserver +8. For mutating requests, the aggregated apiserver runs admission checks. + by default, the namespace lifecycle admission plugin ensures namespaced + resources are created in a namespace that exists in the kube-apiserver -----END----- --> ### Kubernetes Apiserver Authentication and Authorization -A request to an API path that is served by an extension apiserver begins the same way as all API requests: communication to the Kubernetes apiserver. This path already has been registered with the Kubernetes apiserver by the extension apiserver. +A request to an API path that is served by an extension apiserver begins +the same way as all API requests: communication to the Kubernetes apiserver. +This path already has been registered with the Kubernetes apiserver by the extension apiserver. -The user communicates with the Kubernetes apiserver, requesting access to the path. The Kubernetes apiserver uses standard authentication and authorization configured with the Kubernetes apiserver to authenticate the user and authorize access to the specific path. +The user communicates with the Kubernetes apiserver, requesting access to the path. +The Kubernetes apiserver uses standard authentication and authorization configured +with the Kubernetes apiserver to authenticate the user and authorize access to the specific path. -For an overview of authenticating to a Kubernetes cluster, see ["Authenticating to a Cluster"](/docs/reference/access-authn-authz/authentication/). For an overview of authorization of access to Kubernetes cluster resources, see ["Authorization Overview"](/docs/reference/access-authn-authz/authorization/). +For an overview of authenticating to a Kubernetes cluster, see +["Authenticating to a Cluster"](/docs/reference/access-authn-authz/authentication/). +For an overview of authorization of access to Kubernetes cluster resources, see +["Authorization Overview"](/docs/reference/access-authn-authz/authorization/). Everything to this point has been standard Kubernetes API requests, authentication and authorization. @@ -114,50 +144,75 @@ The Kubernetes apiserver now is prepared to send the request to the extension ap ### Kubernetes Apiserver Proxies the Request -The Kubernetes apiserver now will send, or proxy, the request to the extension apiserver that registered to handle the request. In order to do so, it needs to know several things: +The Kubernetes apiserver now will send, or proxy, the request to the extension +apiserver that registered to handle the request. In order to do so, +it needs to know several things: -1. How should the Kubernetes apiserver authenticate to the extension apiserver, informing the extension apiserver that the request, which comes over the network, is coming from a valid Kubernetes apiserver? -2. How should the Kubernetes apiserver inform the extension apiserver of the username and group for which the original request was authenticated? +1. How should the Kubernetes apiserver authenticate to the extension apiserver, + informing the extension apiserver that the request, which comes over the network, + is coming from a valid Kubernetes apiserver? +2. How should the Kubernetes apiserver inform the extension apiserver of the + username and group for which the original request was authenticated? In order to provide for these two, you must configure the Kubernetes apiserver using several flags. #### Kubernetes Apiserver Client Authentication -The Kubernetes apiserver connects to the extension apiserver over TLS, authenticating itself using a client certificate. You must provide the following to the Kubernetes apiserver upon startup, using the provided flags: +The Kubernetes apiserver connects to the extension apiserver over TLS, +authenticating itself using a client certificate. You must provide the +following to the Kubernetes apiserver upon startup, using the provided flags: * private key file via `--proxy-client-key-file` * signed client certificate file via `--proxy-client-cert-file` * certificate of the CA that signed the client certificate file via `--requestheader-client-ca-file` * valid Common Name values (CNs) in the signed client certificate via `--requestheader-allowed-names` -The Kubernetes apiserver will use the files indicated by `--proxy-client-*-file` to authenticate to the extension apiserver. In order for the request to be considered valid by a compliant extension apiserver, the following conditions must be met: +The Kubernetes apiserver will use the files indicated by `--proxy-client-*-file` +to authenticate to the extension apiserver. In order for the request to be considered +valid by a compliant extension apiserver, the following conditions must be met: -1. The connection must be made using a client certificate that is signed by the CA whose certificate is in `--requestheader-client-ca-file`. -2. The connection must be made using a client certificate whose CN is one of those listed in `--requestheader-allowed-names`. +1. The connection must be made using a client certificate that is signed by + the CA whose certificate is in `--requestheader-client-ca-file`. +2. The connection must be made using a client certificate whose CN is one of + those listed in `--requestheader-allowed-names`. -{{< note >}}You can set this option to blank as `--requestheader-allowed-names=""`. This will indicate to an extension apiserver that _any_ CN is acceptable. +{{< note >}} +You can set this option to blank as `--requestheader-allowed-names=""`. +This will indicate to an extension apiserver that _any_ CN is acceptable. {{< /note >}} When started with these options, the Kubernetes apiserver will: 1. Use them to authenticate to the extension apiserver. -2. Create a configmap in the `kube-system` namespace called `extension-apiserver-authentication`, in which it will place the CA certificate and the allowed CNs. These in turn can be retrieved by extension apiservers to validate requests. +2. Create a configmap in the `kube-system` namespace called `extension-apiserver-authentication`, + in which it will place the CA certificate and the allowed CNs. These in turn can be retrieved + by extension apiservers to validate requests. -Note that the same client certificate is used by the Kubernetes apiserver to authenticate against _all_ extension apiservers. It does not create a client certificate per extension apiserver, but rather a single one to authenticate as the Kubernetes apiserver. This same one is reused for all extension apiserver requests. +Note that the same client certificate is used by the Kubernetes apiserver to authenticate +against _all_ extension apiservers. It does not create a client certificate per extension +apiserver, but rather a single one to authenticate as the Kubernetes apiserver. +This same one is reused for all extension apiserver requests. #### Original Request Username and Group -When the Kubernetes apiserver proxies the request to the extension apiserver, it informs the extension apiserver of the username and group with which the original request successfully authenticated. It provides these in http headers of its proxied request. You must inform the Kubernetes apiserver of the names of the headers to be used. +When the Kubernetes apiserver proxies the request to the extension apiserver, +it informs the extension apiserver of the username and group with which the +original request successfully authenticated. It provides these in http headers +of its proxied request. You must inform the Kubernetes apiserver of the names +of the headers to be used. * the header in which to store the username via `--requestheader-username-headers` * the header in which to store the group via `--requestheader-group-headers` * the prefix to append to all extra headers via `--requestheader-extra-headers-prefix` -These header names are also placed in the `extension-apiserver-authentication` configmap, so they can be retrieved and used by extension apiservers. +These header names are also placed in the `extension-apiserver-authentication` configmap, +so they can be retrieved and used by extension apiservers. ### Extension Apiserver Authenticates the Request -The extension apiserver, upon receiving a proxied request from the Kubernetes apiserver, must validate that the request actually did come from a valid authenticating proxy, which role the Kubernetes apiserver is fulfilling. The extension apiserver validates it via: +The extension apiserver, upon receiving a proxied request from the Kubernetes apiserver, +must validate that the request actually did come from a valid authenticating proxy, +which role the Kubernetes apiserver is fulfilling. The extension apiserver validates it via: 1. Retrieve the following from the configmap in `kube-system`, as described above: * Client CA certificate @@ -168,17 +223,28 @@ The extension apiserver, upon receiving a proxied request from the Kubernetes ap * Has a CN in the list of allowed CNs, unless the list is blank, in which case all CNs are allowed. * Extract the username and group from the appropriate headers -If the above passes, then the request is a valid proxied request from a legitimate authenticating proxy, in this case the Kubernetes apiserver. +If the above passes, then the request is a valid proxied request from a legitimate +authenticating proxy, in this case the Kubernetes apiserver. -Note that it is the responsibility of the extension apiserver implementation to provide the above. Many do it by default, leveraging the `k8s.io/apiserver/` package. Others may provide options to override it using command-line options. +Note that it is the responsibility of the extension apiserver implementation to provide +the above. Many do it by default, leveraging the `k8s.io/apiserver/` package. +Others may provide options to override it using command-line options. -In order to have permission to retrieve the configmap, an extension apiserver requires the appropriate role. There is a default role named `extension-apiserver-authentication-reader` in the `kube-system` namespace which can be assigned. +In order to have permission to retrieve the configmap, an extension apiserver +requires the appropriate role. There is a default role named `extension-apiserver-authentication-reader` +in the `kube-system` namespace which can be assigned. ### Extension Apiserver Authorizes the Request -The extension apiserver now can validate that the user/group retrieved from the headers are authorized to execute the given request. It does so by sending a standard [SubjectAccessReview](/docs/reference/access-authn-authz/authorization/) request to the Kubernetes apiserver. +The extension apiserver now can validate that the user/group retrieved from +the headers are authorized to execute the given request. It does so by sending +a standard [SubjectAccessReview](/docs/reference/access-authn-authz/authorization/) +request to the Kubernetes apiserver. -In order for the extension apiserver to be authorized itself to submit the `SubjectAccessReview` request to the Kubernetes apiserver, it needs the correct permissions. Kubernetes includes a default `ClusterRole` named `system:auth-delegator` that has the appropriate permissions. It can be granted to the extension apiserver's service account. +In order for the extension apiserver to be authorized itself to submit the +`SubjectAccessReview` request to the Kubernetes apiserver, it needs the correct permissions. +Kubernetes includes a default `ClusterRole` named `system:auth-delegator` that +has the appropriate permissions. It can be granted to the extension apiserver's service account. ### Extension Apiserver Executes @@ -187,7 +253,8 @@ If the `SubjectAccessReview` passes, the extension apiserver executes the reques ## Enable Kubernetes Apiserver flags -Enable the aggregation layer via the following `kube-apiserver` flags. They may have already been taken care of by your provider. +Enable the aggregation layer via the following `kube-apiserver` flags. +They may have already been taken care of by your provider. --requestheader-client-ca-file= --requestheader-allowed-names=front-proxy-client @@ -204,20 +271,44 @@ The Kubernetes apiserver has two client CA options: * `--client-ca-file` * `--requestheader-client-ca-file` -Each of these functions independently and can conflict with each other, if not used correctly. - -* `--client-ca-file`: When a request arrives to the Kubernetes apiserver, if this option is enabled, the Kubernetes apiserver checks the certificate of the request. If it is signed by one of the CA certificates in the file referenced by `--client-ca-file`, then the request is treated as a legitimate request, and the user is the value of the common name `CN=`, while the group is the organization `O=`. See the [documentation on TLS authentication](/docs/reference/access-authn-authz/authentication/#x509-client-certs). -* `--requestheader-client-ca-file`: When a request arrives to the Kubernetes apiserver, if this option is enabled, the Kubernetes apiserver checks the certificate of the request. If it is signed by one of the CA certificates in the file reference by `--requestheader-client-ca-file`, then the request is treated as a potentially legitimate request. The Kubernetes apiserver then checks if the common name `CN=` is one of the names in the list provided by `--requestheader-allowed-names`. If the name is allowed, the request is approved; if it is not, the request is not. - -If _both_ `--client-ca-file` and `--requestheader-client-ca-file` are provided, then the request first checks the `--requestheader-client-ca-file` CA and then the `--client-ca-file`. Normally, different CAs, either root CAs or intermediate CAs, are used for each of these options; regular client requests match against `--client-ca-file`, while aggregation requests match against `--requestheader-client-ca-file`. However, if both use the _same_ CA, then client requests that normally would pass via `--client-ca-file` will fail, because the CA will match the CA in `--requestheader-client-ca-file`, but the common name `CN=` will **not** match one of the acceptable common names in `--requestheader-allowed-names`. This can cause your kubelets and other control plane components, as well as end-users, to be unable to authenticate to the Kubernetes apiserver. - -For this reason, use different CA certs for the `--client-ca-file` option - to authorize control plane components and end-users - and the `--requestheader-client-ca-file` option - to authorize aggregation apiserver requests. +Each of these functions independently and can conflict with each other, +if not used correctly. + +* `--client-ca-file`: When a request arrives to the Kubernetes apiserver, + if this option is enabled, the Kubernetes apiserver checks the certificate + of the request. If it is signed by one of the CA certificates in the file referenced by + `--client-ca-file`, then the request is treated as a legitimate request, + and the user is the value of the common name `CN=`, while the group is the organization `O=`. + See the [documentation on TLS authentication](/docs/reference/access-authn-authz/authentication/#x509-client-certs). +* `--requestheader-client-ca-file`: When a request arrives to the Kubernetes apiserver, + if this option is enabled, the Kubernetes apiserver checks the certificate of the request. + If it is signed by one of the CA certificates in the file reference by `--requestheader-client-ca-file`, + then the request is treated as a potentially legitimate request. The Kubernetes apiserver then + checks if the common name `CN=` is one of the names in the list provided by `--requestheader-allowed-names`. + If the name is allowed, the request is approved; if it is not, the request is not. + +If _both_ `--client-ca-file` and `--requestheader-client-ca-file` are provided, +then the request first checks the `--requestheader-client-ca-file` CA and then the +`--client-ca-file`. Normally, different CAs, either root CAs or intermediate CAs, +are used for each of these options; regular client requests match against `--client-ca-file`, +while aggregation requests match against `--requestheader-client-ca-file`. However, +if both use the _same_ CA, then client requests that normally would pass via `--client-ca-file` +will fail, because the CA will match the CA in `--requestheader-client-ca-file`, +but the common name `CN=` will **not** match one of the acceptable common names in +`--requestheader-allowed-names`. This can cause your kubelets and other control plane components, +as well as end-users, to be unable to authenticate to the Kubernetes apiserver. + +For this reason, use different CA certs for the `--client-ca-file` +option - to authorize control plane components and end-users - and the `--requestheader-client-ca-file` option - to authorize aggregation apiserver requests. {{< warning >}} -Do **not** reuse a CA that is used in a different context unless you understand the risks and the mechanisms to protect the CA's usage. +Do **not** reuse a CA that is used in a different context unless you understand +the risks and the mechanisms to protect the CA's usage. {{< /warning >}} -If you are not running kube-proxy on a host running the API server, then you must make sure that the system is enabled with the following `kube-apiserver` flag: +If you are not running kube-proxy on a host running the API server, +then you must make sure that the system is enabled with the following +`kube-apiserver` flag: --enable-aggregator-routing=true @@ -276,6 +367,8 @@ spec: ## {{% heading "whatsnext" %}} -* [Set up an extension api-server](/docs/tasks/extend-kubernetes/setup-extension-api-server/) to work with the aggregation layer. -* For a high level overview, see [Extending the Kubernetes API with the aggregation layer](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/). +* [Set up an extension api-server](/docs/tasks/extend-kubernetes/setup-extension-api-server/) + to work with the aggregation layer. +* For a high level overview, see + [Extending the Kubernetes API with the aggregation layer](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/). * Learn how to [Extend the Kubernetes API Using Custom Resource Definitions](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/).