Add docs for configuring kubelet credential provider plugins #25226
Conversation
|
Deploy preview for kubernetes-io-vnext-staging processing. Building with commit 924acf8 https://app.netlify.com/sites/kubernetes-io-vnext-staging/deploys/5fc8372a3c7a790008a61bcf |
k8s-ci-robot
added
the
size/L
k8s-ci-robot
added
language/en
|
/milestone 1.20 |
k8s-ci-robot
added
the
sig/cloud-provider
k8s-ci-robot
added
the
sig/node
|
|
||
| ## {{% heading "prerequisites" %}} | ||
|
|
||
| * kubelet on v1.20 or later. If on v1.20, ensure the alpha feature gate `KubeletCredentialProviders` is on. |
There was a problem hiding this comment.
KubeletCredentialProviders needs to be added to the feature gate list content/en/docs/reference/command-line-tools-reference/feature-gates/ with a short description describing the feature
There was a problem hiding this comment.
How about:
* The kubelet image credential provider is introduced in 1.20 as an alpha feature.
As with other alpha features, a feature gate `KubeletCredentialProviders` must be
explicitly enabled for the feature to work
Not a blocker though. Just trying to ensure the information will not become outdated soon. For example, we don't know if the feature will stay in alpha on v1.21.
There was a problem hiding this comment.
Great call @tengqm, will update with your suggestion
| 1. API calls to a cloud provider service are required to retreive authentication information for a registry. | ||
| 2. Credentials have short expiration times and requesting new credentials frequently is required. | ||
| 3. Storing registry credentials on disk or in image pull Secrets is not acceptable. |
There was a problem hiding this comment.
nits:
- add an empty line before line 19
- avoid explicit numbers in ordered list, always use
1.
| This guide demonstrates how to configure the kubelet's image credential provider plugin mechanism. | ||
|
|
||
| {{< note >}} | ||
| This capabilitiy will eventually replace built-in logic in the kubelet to serve registry credentials for Azure, AWS and GCP. |
There was a problem hiding this comment.
this is a bit forward looking, we'd better avoid statements about future in the doc.
| * `--image-credential-provider-config` - the path to the credential provider plugin config file. | ||
| * `--image-credential-provider-bin-dir` - the path to the directory where credential provider plugin binaries are located. |
There was a problem hiding this comment.
Better document the equivalent kubelet config field names here as well. We are seemingly moving away from command line flags anyway.
There was a problem hiding this comment.
@andrewsykim are there corresponding config field names, or are these only flags?
There was a problem hiding this comment.
We did not add config fields for these, which I think has been expected for container runtime options. For example, none of the flags in ContainerRuntimeOptions have config field equivalents:
| * `--image-credential-provider-config` - the path to the credential provider plugin config file. | ||
| * `--image-credential-provider-bin-dir` - the path to the directory where credential provider plugin binaries are located. | ||
|
|
||
| If you are on v1.20, ensure the feature gate `KubeletCredentialProviders` is enabled via the `--feature-gates` flag. |
There was a problem hiding this comment.
This prerequisite is already mentioned on line 31
|
|
||
| ## {{% heading "prerequisites" %}} | ||
|
|
||
| * kubelet on v1.20 or later. If on v1.20, ensure the alpha feature gate `KubeletCredentialProviders` is on. |
There was a problem hiding this comment.
How about:
* The kubelet image credential provider is introduced in 1.20 as an alpha feature.
As with other alpha features, a feature gate `KubeletCredentialProviders` must be
explicitly enabled for the feature to work
Not a blocker though. Just trying to ensure the information will not become outdated soon. For example, we don't know if the feature will stay in alpha on v1.21.
| - "*.dkr.ecr-fips.*.amazonaws.com" | ||
| - "*.dkr.ecr.us-iso-east-1.c2s.ic.gov" | ||
| - "*.dkr.ecr.us-isob-east-1.sc2s.sgov.gov" | ||
| defaultCacheDuration: 12h |
There was a problem hiding this comment.
| defaultCacheDuration: 12h | |
| defaultCacheDuration: "12h" |
| The `matchImages` field for each credenetial provider is used by the kubelet to determine whether a plugin should be invoked for a given image that a Pod is using. | ||
| Each entry in `matchImages` is a pattern which can optionally contain a port and a path. Globs can be used in the domain, but not in the port or the path. Globs are | ||
| supported as subdomains like '\*.k8s.io' or 'k8s.\*.io', and top-level domains such as 'k8s.\*'. Matching partial subdomains like 'app\*.k8s.io' is also supported. Each | ||
| glob can only match a single subdomain segment, so '\*.io' does NOT match '\*.k8s.io'. |
There was a problem hiding this comment.
Please consider wrap the long lines properly so that future changes can be easily tracked.
|
|
||
| The `matchImages` field for each credenetial provider is used by the kubelet to determine whether a plugin should be invoked for a given image that a Pod is using. | ||
| Each entry in `matchImages` is a pattern which can optionally contain a port and a path. Globs can be used in the domain, but not in the port or the path. Globs are | ||
| supported as subdomains like '\*.k8s.io' or 'k8s.\*.io', and top-level domains such as 'k8s.\*'. Matching partial subdomains like 'app\*.k8s.io' is also supported. Each |
There was a problem hiding this comment.
| supported as subdomains like '\*.k8s.io' or 'k8s.\*.io', and top-level domains such as 'k8s.\*'. Matching partial subdomains like 'app\*.k8s.io' is also supported. Each | |
| supported as subdomains like `*.k8s.io` or `k8s.*.io`, and top-level domains such as | |
| `k8s.*`. Matching partial subdomains like `app*.k8s.io` is also supported. Each |
The point here is that by wrapping these literals with backtiqs, readers can easily identify what are the patterns with NO confusion. Having backslashes in the text can thus considered harmful.
There was a problem hiding this comment.
Good call, will update
|
|
||
| A match exists between an image and a matchImage entry when all of the below are true: | ||
| * Both contain the same number of domain parts and each part matches. | ||
| * The URL path of an imageMatch must be a prefix of the target image URL path. |
| * 123456789.dkr.ecr.us-east-1.amazonaws.com | ||
| * \*.azurecr.io | ||
| * gcr.io | ||
| * \*.\*.registry.io | ||
| * foo.registry.io:8080/path |
There was a problem hiding this comment.
plse consider using backtiqs for these string literals.
There was a problem hiding this comment.
@andrewsykim, my take on this documentation is that it is closer to a conceptual overview than a task that the reader could step through.
I am not sure where the best home for this might be. I have a suggestion, though:
https://kubernetes.io/docs/concepts/containers/image-credential-plugins/
Would you be willing to move this documentation there and switch the content_type to concept?
| ## {{% heading "prerequisites" %}} | ||
|
|
||
| * The kubelet image credential provider is introduced in v1.20 as an alpha feature. As with other alpha features, | ||
| a feature gate `KubeletCredentialProviders` must be explicitly enabled for the feature to work |
There was a problem hiding this comment.
nit: link to the list of feature gates.
There was a problem hiding this comment.
Where does the reader enable that feature gate? Just for the kubelet?
|
|
||
| ## Installing Plugins on Nodes | ||
|
|
||
| A credential provider plugin is an exectuable binary that will be run by the kubelet. Ensure that the plugin binary exists on |
There was a problem hiding this comment.
I noticed a spelling issue:
| A credential provider plugin is an exectuable binary that will be run by the kubelet. Ensure that the plugin binary exists on | |
| A credential provider plugin is an executable binary that will be run by the kubelet. Ensure that the plugin binary exists on |
| * `--image-credential-provider-config` - the path to the credential provider plugin config file. | ||
| * `--image-credential-provider-bin-dir` - the path to the directory where credential provider plugin binaries are located. | ||
|
|
||
| ### Credential Provider Configuration File |
There was a problem hiding this comment.
As this is a task page, a better heading might be
| ### Credential Provider Configuration File | |
| ### Configure a kubelet credential provider |
| ### Credential Provider Configuration File | ||
|
|
||
| The configuration file passed into `--image-credential-provider-config` is read by the kubelet to determine which exec plugins | ||
| should be invoked for which container images. Here's an example configuration file you may end up using if you are using the ECR-based plugin: |
There was a problem hiding this comment.
Consider expanding ECR or hyperlinking to a page that explains what it is.
| defaultCacheDuration: "12h" | ||
| apiVersion: credentialprovider.kubelet.k8s.io/v1alpha1 | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Should this task list an AWS EC2 instance as a prerequisite? Alternatively, is there a different credential mechanism that we can explain, perhaps something more generic?
There was a problem hiding this comment.
Trying to use a real example in the docs instead of dummy values. I think AWS ECR is a good reference because you can enable it on non-ec2 instance as long as there are credentials on disk.
|
|
||
| Consult the plugin implementors to determine what set of arguments and environment variables are required for a given plugin. | ||
|
|
||
| #### Match Images |
There was a problem hiding this comment.
| #### Match Images | |
| #### Configure image matching |
| and top-level domains such as `k8s.*`. Matching partial subdomains like `app*.k8s.io` is also supported. Each glob can only match | ||
| a single subdomain segment, so `*.io` does NOT match `*.k8s.io`. | ||
|
|
||
| A match exists between an image and a matchImage entry when all of the below are true: |
There was a problem hiding this comment.
| A match exists between an image and a matchImage entry when all of the below are true: | |
| A match exists between an image name and a `matchImage` entry when all of the below are true: |
| * The URL path of match image must be a prefix of the target image URL path. | ||
| * If the imageMatch contains a port, then the port must match in the image as well. | ||
|
|
||
| Some example values of `matchImages` are: |
There was a problem hiding this comment.
| Some example values of `matchImages` are: | |
| Some example values of `matchImages` patterns are: |
| A credential provider plugin is an exectuable binary that will be run by the kubelet. Ensure that the plugin binary exists on | ||
| every node in your cluster and stored in a known directory. The directory will be required later when configuring kubelet flags. | ||
|
|
||
| ## Configuring Kubelet |
There was a problem hiding this comment.
| ## Configuring Kubelet | |
| ## Configure the kubelet |
|
|
||
| ## Configuring Kubelet | ||
|
|
||
| ### Flags |
|
Just a friendly reminder that Docs deadline to get this merged in is in two days, Dec. 2nd |
@sftim I had initially added this to "Concepts" but you suggested moving it to "Tasks" in #24929 (review) 😛 -- I think it makes a lot of sense though. I can move it back but I'll need a day or two to work it out. Sorry for the delay. |
content/en/docs/tasks/kubelet-credential-provider/kubelet-credential-provider.md
Show resolved
Hide resolved
| Starting from Kubernetes v1.20, the kubelet can dynamically retrieve credentials for a container image registry | ||
| using exec plugins. The kubelet and the exec plugin communicate through stdio (stdin, stdout, and stderr) using | ||
| Kubernetes versioned APIs. These plugins allow the kubelet to request credentials for a container registry dynamically | ||
| as oppposed to storing static credentials on disk. For example, the plugin may talk to a local metadata server to retrieve |
There was a problem hiding this comment.
| as oppposed to storing static credentials on disk. For example, the plugin may talk to a local metadata server to retrieve | |
| as opposed to storing static credentials on disk. For example, the plugin may talk to a local metadata server to retrieve |
|
|
||
| * API calls to a cloud provider service are required to retreive authentication information for a registry. | ||
| * Credentials have short expiration times and requesting new credentials frequently is required. | ||
| * Storing registry credentials on disk or in image pull Secrets is not acceptable. |
There was a problem hiding this comment.
| * Storing registry credentials on disk or in image pull Secrets is not acceptable. | |
| * Storing registry credentials on disk or in imagePullSecrets is not acceptable. |
|
|
||
| You may be interested in using this capabilitiy if any of the below are true: | ||
|
|
||
| * API calls to a cloud provider service are required to retreive authentication information for a registry. |
There was a problem hiding this comment.
| * API calls to a cloud provider service are required to retreive authentication information for a registry. | |
| * API calls to a cloud provider service are required to retrieve authentication information for a registry. |
| as oppposed to storing static credentials on disk. For example, the plugin may talk to a local metadata server to retrieve | ||
| short-lived credentials for an image that is being pulled by the kubelet. | ||
|
|
||
| You may be interested in using this capabilitiy if any of the below are true: |
There was a problem hiding this comment.
| You may be interested in using this capabilitiy if any of the below are true: | |
| You may be interested in using this capability if any of the below are true: |
| * `--image-credential-provider-config` - the path to the credential provider plugin config file. | ||
| * `--image-credential-provider-bin-dir` - the path to the directory where credential provider plugin binaries are located. |
There was a problem hiding this comment.
@andrewsykim are there corresponding config field names, or are these only flags?
| ```yaml | ||
| kind: CredentialProviderConfig | ||
| apiVersion: kubelet.config.k8s.io/v1alpha1 | ||
| providers: |
There was a problem hiding this comment.
I've found it helpful to annotate config and request/response documentation with the API docs, e.g. https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#request
This is a great place to explain what each field means, what it does, etc
There was a problem hiding this comment.
Good call, will updat
|
|
||
| #### Match Images | ||
|
|
||
| The `matchImages` field for each credenetial provider is used by the kubelet to determine whether a plugin should be invoked |
There was a problem hiding this comment.
| The `matchImages` field for each credenetial provider is used by the kubelet to determine whether a plugin should be invoked | |
| The `matchImages` field for each credential provider is used by the kubelet to determine whether a plugin should be invoked |
Yes, can someone from SIG docs recommend where a document like this should live on the doc site? |
|
@sftim I'm thinking that this feature actually needs both a "concepts" and "tasks" page, as well as a developer-focused page for plugin authors. I'd like to keep this PR scoped to just the tasks page but I will follow-up to add the others before this feature goes beta. |
I think this works, but I will wait for Tim/others to chime in. Maybe opening an issue where this can be tracked could help make sure this doesn't get lost could help as well. |
Hi, I'm a bit too busy at the moment - please feel free to get reviews elsewhere, and to dismiss / ignore any earlier reviews of mine if superseded. |
|
@andrewsykim In the interests of time, and after reading through this doc, I think it's well-placed in |
|
Seconding @celestehorgan's comment. I think the content is sufficient to enable early adopters of 1.20 to leverage the new enhancement as-is, which is the most important part of getting docs in before the actual release. /lgtm |
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. Git tree hash: 40dde3e30651b8e6e256013b52cee9889f975311
|
Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>
Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>
k8s-ci-robot
removed
the
lgtm
Title updated! |
|
Additional 👍 to
One good possibility might be |
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. Git tree hash: efa85f01d02043d7a215ca98a8c30b1f49d97b67
|
|
Hi @liggitt @cheftako pinging for tech review again. Please note that your concern with documentation on the plugin will be addressed when the feature graduates to beta as noted here #25226 (review) and #25226 (comment) Once we have tech review / lgtm, this should be good to merge. |
|
concerns with needing more docs will be addressed when features goes to beta /approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: annajung The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
Signed-off-by: Andrew Sy Kim kim.andrewsy@gmail.com
Add docs for configuring kubelet credential provider plugin. Replaces #24929 since that was targeting the wrong branch.