Kubelet server certificate bootstrap and rotation #602
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| # Kubelet Server Certificate Bootstrap & Rotation | ||
|
|
||
| ## Overview | ||
|
|
||
| Currently, when a kubelet first starts, it generates a self signed | ||
| certificate/key pair that is used for accepting incoming TLS connections. This | ||
| proposal covers a process for generating a key locally and then issuing a | ||
| Certificate Signing Request to the cluster API server to get an associated | ||
| certificate signed by the cluster Certificate Authority. Also, as certificates | ||
| approach expiration, the same mechanism will be used to request an updated | ||
| certificate. | ||
|
|
||
| ## Use Cases | ||
|
|
||
| 1. Kubelet starts for the first time on a new node. | ||
| 2. Kubelet certificate approaches expiration. | ||
|
|
||
| ## Proposed Design | ||
|
|
||
| ### During Kubelet Boot Sequence | ||
|
|
||
| 1. Look on disk for an existing cert/key pair. | ||
| 1. If there is an existing cert/key pair, load them. | ||
| 1. If there is no existing cert/key pair, at the point where the kubelet | ||
| currently generates a key and certificate, the kubelet will: | ||
| - generate a key | ||
| - create a certificate signing request | ||
| - request a signed certificate from the API server | ||
| - wait for a response | ||
| - Store the new cert/key pairs in the directory specified by `--cert-dir`, | ||
| or the default value if it is unspecified. | ||
|
|
||
| ### As Expiration Approaches | ||
|
|
||
| Rotating a Kubelet server certificate will work by generating a new private key, | ||
|
There was a problem hiding this comment. Any reason it's regenerating a new private key rather then reusing the old one? There was a problem hiding this comment. generally, you want young keys... the older a key, the longer the exposure to factoring. if you have an auto-renewal system, using a new key each time is good. There was a problem hiding this comment. Optional or required? I suspect optional, but I'd like it called out as an optional feature at the top level. There was a problem hiding this comment. Do you mean certificate rotation should be optional? Currently it is gated by a feature flag due to it's alpha status. Would you like it gated by a command line parameter and rotation of expiring certificates would be an optional feature of a kubelet? There was a problem hiding this comment. even once the rotation feature is no longer alpha, not all deployments will use cert rotation. I would expect to be able to continue providing a cert/key explicitly to the kubelet and have it simply use it. |
||
| issuing a new Certificate Signing Request to the API Server, safely updating the | ||
| cert/key pair on disk, begin using the new cert/key pair. | ||
|
|
||
| 1. Store the new cert/key pairs in the directory specified by `--cert-dir`, or the | ||
| default value if it is unspecified. This will allow the kubelet to have a | ||
| place for storing the multiple cert/key pairs that it might have available at | ||
| any given moment (because of a rotation in progress). | ||
| - When cert/key files are specified with the `--tls-cert-file` and | ||
| `--tls-private-key-file`: | ||
| - These values be used only if there are no updated cert/key pairs due | ||
|
There was a problem hiding this comment. Having trouble parsing this section. Does this imply that the kubelet will try to request new serving certs even if you've configured them through flags? There was a problem hiding this comment. I'd expect either an explicitly specified cert or an auto-rotated cert, not both There was a problem hiding this comment. These were pre-existing flags and I'm trying to find a way to nicely incorporate their behavior. The way I see this, section to to use this cert/key pair if it is there, and be able to move on to rotation when the time is right. I could do as Jordan suggests, but it makes it harder to transition an existing cluster. If someone has a cluster, with this cert/key specified, then these are the valid authentication mechanisms for connecting to the API server in order to do a rotation. However, once there is a rotation, that cert/key pair will be newer, and these parameters will be ignored. Analogous to how bootstrapping works, a cert/key pair specified in the kube config file is valid, as long as there is no more up to date cert/key pair. |
||
| to rotation | ||
| 1. There will be a feature gate to enable certificate bootstrapping and | ||
| rotation, `RotateKubeletServerCertificate` | ||
| 1. Centralize certificate access within the kubelet code to the | ||
| CertificateManager. The CertificateManager will be responsible for: | ||
| - Providing the correct certificate to use for establishing TLS connections. | ||
|
There was a problem hiding this comment. This is the CA bundle used to verify a connection to the There was a problem hiding this comment. This is the certificate that is used for any incoming connections to port 10250. CAs are not affected by this proposal. |
||
| - Generating new private keys and requesting new certificates when the | ||
| current certificate approaches expiry. | ||
| - Since certificates can rotate at any time, all other parts of the kubelet | ||
| should ask the CertificateManager for the correct certificate each time a | ||
| certificate is used. No certificate caching except by the | ||
|
There was a problem hiding this comment. How will you prevent the use of the "normal" connection caches we have in the client? There was a problem hiding this comment. On the client side the certificate manager overrides the restclient.Config GetClientCertificate method[0]. [0] https://github.com/kubernetes/kubernetes/pull/41912/files#diff-e0160f895346669c2e4495acfd858ea7R466 There was a problem hiding this comment. making the client certificate dynamic is problematic for the transport cache at https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/transport/cache.go#L41 There was a problem hiding this comment. Ah I see that #41912 actually added the GetClientCertificate field to the TLS config. Does it work for the cache to ignore configs that set this new field? I'm under the impression that the kubelet is a lot better about keeping a centralized client instead of constantly creating new ones like kubectl. There was a problem hiding this comment. yeah, those have to be uncacheable. I'm not sure how many transports get created with a call to something like |
||
| CertificateManager. | ||
| - TLS connections should prefer to set the | ||
| [`GetCertificate`](https://golang.org/pkg/crypto/tls/#Config) and | ||
| [`GetClientCertificate`](https://golang.org/pkg/crypto/tls/#Config) | ||
| callbacks so that the connection dynamically requests the certificate | ||
| as needed. | ||
| - Recovering from kubelet crashes or restarts that occur while certificate | ||
| transitions are in flight (request issued, but not yet signed, etc) | ||
| 1. The RBAC role for nodes will be updated to allow nodes to request new | ||
| certificates. | ||
| 1. Have the CertificateManager repeat the CSR process as certificate expiration | ||
| approaches. | ||
| - New certificates will be requested when the configured duration threshold | ||
|
There was a problem hiding this comment. Is this actually configurable through a command line flag or something? There was a problem hiding this comment. Duration is chosen by the Certificate Request Signing API. There is currently no flag for changing the duration that will be assigned, but I've been thinking of adding one to facilitate testing. There was a problem hiding this comment. There is now a flag ( |
||
| has been exceeded. | ||
| - Crash safe file structure | ||
| - Private key for requests in flight are held in memory. In case of | ||
| interruption, they will be abandoned. | ||
| - When the corresponding signed certificate is received, the cert/key | ||
| pair will be written to a single file, eg. | ||
| kubelet-server-<timestamp>.pem. | ||
| - Write kubelet-server-updated.pem symlink to point to the new cert/key | ||
| pair. | ||
| - Move kubelet-server-updated.pem to kubelet-server-current.pem | ||
|
|
||
| ### Certificate Approval | ||
|
|
||
| With the kubelet requesting certificates be signed as part of is boot sequences, | ||
| and on an ongoing basis, it makes cluster administration easier if certificate | ||
| signing requests from the kubelet were automatically approved. A full solution | ||
| to this will require authenticating the kubelet making the request in some way, | ||
| and will be covered by a [future design] | ||
| (https://github.com/kubernetes/kubernetes/issues/45030). | ||
|
|
||
| ## Future Work | ||
|
|
||
| 1. Currently the Certificate Request Signing controller is hard coded to issue | ||
| certificates for 1 year. Making this a parameter that can be set quite short | ||
| during end to end testing would enable easy certificate rotation testing. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it doesn't get approved, what will it do? What if it doesn't get an answer?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't proceed.