KEP: Arbitrary image pull secrets

[sashayakovtseva]

This KEP emerged from kubernetes/kubernetes#79803

[k8s-ci-robot]

Welcome @sashayakovtseva!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @sashayakovtseva. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sashayakovtseva]

/sig-node

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sashayakovtseva
To complete the pull request process, please assign derekwaynecarr
You can assign the PR to them by writing /assign @derekwaynecarr in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/sig-node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sashayakovtseva]

/assign @derekwaynecarr

[dims]

@sashayakovtseva have you seen this? #1137

[sashayakovtseva]

@dims Nope, just took a look. As I understand #1137 is about connecting to kubernetes as a client, kubeconfig and etc, while here I talk about image pull credentials and kubelet component.

[Random-Liu]

/ok-to-test

[Random-Liu]

We may not want to define the schema based on AuthConfig.

CRI is a Kubelet internal API, and we can't guarantee that we won't change or extend AuthConfig in the future. If we changed it, how do we handle the version skew?

[sashayakovtseva]

Another option would be to pass pull secrets as map[string]string or smth and let CRI decide what to do, but I am not sure that is better than AuthConfig mapping.

[Random-Liu]

@sashayakovtseva I'm a bit confused. Does the AuthConfig object meet your requirement?

If it does, it should be equivalent to today's docker secret https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials. Kubelet just converts docker secret into AuthConfig.

If AuthConfig works for you, you just need to fill in the value into corresponding fields in the docker secret, and then the docker secret should just work for you.

[sashayakovtseva]

@Random-Liu AuthConfig can be used. We are using access tokens to pull private images, so AuthConfig.IdentityToken is the proper field to use IMO.

And yes, the workaround you suggest partially works. Here are the issues that should be addressed:

1. Forcing any CRI to use secret of type kubernetes.io/dockerconfigjson doesn't match with idea of abstracting runtime with CRI. It also requires username which we don't need because of access tokens instead of username/password pair.
2. I could not find where AuthConfig.ServerAddress is set on kubelet side. Without that it is not possible to use on-prem image registries which we want to leverage.

That is why even though with a workaround it is possible to pull a private image from default registry, I still believe this should be changed on kubelet side to fully enable runtime abstraction.

[Random-Liu]

It is in the keyring https://github.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/keyring.go#L66,
and kubelet does pass it to CRI https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_image.go#L63.

I'm totally in to a more generic credential format in the Kubernetes api, instead of kubernetes.io/dockerconfigjson. However, we need a use case with more representativeness to define the new API, so we are waiting for a use case that can't be covered by today's kubernetes.io/dockerconfigjson. :)

[Random-Liu]

@sashayakovtseva Hm, I think you are right!

However, I don't understand why we don't support ServerAddress, IdentityToken and RegistryToken in the first place...I think we should fix it.

Anyway, if this is the case, there are 2 options:

1. Fix today's kubernetes.io/dockerconfigjson, and add full docker AuthConfig support.
2. Define a new format, and based on your proposal, it will just be equivalent to docker AuthConfig.

If so, I prefer 1). :)

Need input from @kubernetes/sig-auth-api-reviews

[sashayakovtseva]

@Random-Liu IMO (1) should be a temporarily fix unless Kubernetes wishes to fully support only Docker containers even though there will be other CRI implementations.
I understand that technically this will work, but that is really awkward to ask e.g. Singularity users to create Docker secret to pull private images...

[Random-Liu]

I want 2) in the long term, but not sure whether we can define the new format correctly at the moment. Thus I prefer 1) for now.

If @kubernetes/sig-auth-api-reviews thinks that the AuthConfig format is sufficient and generic enough for Kubernetes, I'm totally fine with 2). :)

[Random-Liu]

BTW, the ServerAddress should be supported anyway to handle the case mentioned in containerd/cri#1227 (comment)

[yujuhong]

+1 to @Random-Liu's suggestion.

I'd like to API to look more generic as well, but given what we have today, it's preferred if we can easily extend the existing API to support your case. If there are other use cases that cannot be easily satisfied, we can introduce a new type.

[k8s-ci-robot]

@sashayakovtseva: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-enhancements-verify	ce93e19	link	/test pull-enhancements-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[derekwaynecarr]

If this was discussed in SIG Node while I was out, I apologize, but I agree with @Random-Liu . I would be interested if @mrunalp is aware of other credential formats that have been brought forward.

[derekwaynecarr]

I think @Random-Liu addressed the answer to point (2), I am missing where (1) requires a username. It's possible I am missing something. Either way, I am in favor of supporting a generic credential format and/or secret type.

[derekwaynecarr]

@mrunalp are there other credential formats that have been brought forward that I may be missing?

[derekwaynecarr]

if the desire is to support alternative secret types, I think that is potentially a fine thing to do, but I am curious how we know if the configured runtime would now how to manage them. It would seem to me they would still require some type of schema to identify their type.

[sashayakovtseva]

@derekwaynecarr Regarding point 1. Consider pull secret creation via CLI:

$ kubectl create secret docker-registry regcred --docker-server=cloud.sylabs.io  --docker-password=my-secret-token
Error: required flag(s) "docker-username" not set

Even though cloud.sylabs.io needs only token, docker-registry secret requires username to be set. Of course, some kind of stub can be used here, but again, that is generic at all.

And of course, we can circumvent it with creating docker-registry secret manually like so:

$ echo '{"auths":{"cloud.sylabs.io":{"password":"my-secret-token"}}}' | base64
eyJhdXRocyI6eyJjbG91ZC5zeWxhYnMuaW8iOnsicGFzc3dvcmQiOiJteS1zZWNyZXQtdG9rZW4ifX19Cg==

$ cat > secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: regcred
data:
  .dockerconfigjson: eyJhdXRocyI6eyJjbG91ZC5zeWxhYnMuaW8iOnsicGFzc3dvcmQiOiJteS1zZWNyZXQtdG9rZW4ifX19Cg==
type: kubernetes.io/dockerconfigjson
EOF

$  kubectl apply -f secret.yaml

But that is not convenient for the users.
Also I have not checked yet whether this manually created secret will work at all, will do so after the weekend.

[mikedanese]

Somewhat related to kubernetes/kubernetes#68810

[fejta-bot]

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

[sashayakovtseva]

According to discussion with @Random-Liu this KEP looks reasonable.
I wonder what are the next steps from my side to get it approved or updated or whatever :)

[Random-Liu]

@sashayakovtseva I think if we plan to fix existing docker authconfig, I think we just need a bug report and fix it.

If we plan to have a new registry credential type, we need to go through Kubernetes api review. https://github.com/kubernetes/community/blob/master/sig-architecture/api-review-process.md

[sashayakovtseva]

@Random-Liu Do I have to write that bug report? If so, to the k8s repository?

If we plan to have a new registry credential type, we need to go through Kubernetes api review.
Who is the one to decide that?

Sorry, but I don't really understand what is the action I have to take to proceed with this KEP :)
Should I wait? Should I submit smth else?

[smarterclayton]

I don’t see a reason to make the secret opaque. We can always change the auth config spec or introduce a new secret type - opaque secrets are against our general approach to defining points of consistency between container runtimes.

I would be supportive of letting container runtimes interpret other information about a pod in order to decide on secret values, but we’d need to be more concrete in this proposal about what that actually means.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
• Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.