ImagePullSecrets doesn't work with registry mirror.

[markx]

It's mentioned in https://containerd.io/releases/#deprecated-config-properties that
[plugins."io.containerd.grpc.v1.cri".registry] auths will be removed on v2.1, and the replacement is to use ImagePullSecrets.

However ImagePullSecrets doesn't work with mirrors, as kubelet only passes containerd the creds for the original host of the pulled image.

Is there plan to add a replacement that works with mirrors?
Is #8228 the solution?
Is there still plan to add creds in host config, as suggested in https://github.com/containerd/containerd/blob/main/core/remotes/docker/config/hosts.go#L57

[dmcgowan]

However ImagePullSecrets doesn't work with mirrors, as kubelet only passes containerd the creds for the original host of the pulled image.

Is there a related KEP for improving this, user owned secrets really is the best solution for registry authentication.

Is there plan to add a replacement that works with mirrors?

Yes, the replacement was not built for 2.0 so we did not remove

Is #8228 the solution?

A proposal for a solution, we haven't defined a plugin interface there yet. Ideally such a solution could also potentially go along with Kubelet change to avoid passing the wrong, or unasked for, credentials over the CRI api with the image pull command. Using secret stores is the better solution so we aren't forced to keep credentials anywhere in plaintext or held in memory longer than they are needed.

Is there still plan to add creds in host config, as suggested in https://github.com/containerd/containerd/blob/main/core/remotes/docker/config/hosts.go#L57

Secrets will not be added in the host config, the comment mentions configurations for credentials though. This could include stuff like the domain or username to request a secret for as well as possibly which plugin to use to get the secret from.

[markx]

Thanks for the quick response, Derek!

Yes, the replacement was not built for 2.0 so we did not remove

Great! So this is already on the radar and there is plan to support. My concern was that this is somehow overlooked, or even worse, that containerd suddenly decides to drop the mirror feature.

Sounds like it's safe to assume that, these settings will stay as long as the replacement is not provided, potentially beyond 2.1?
If this is the case, we can continue using the existing auths until a fully-working replacement is provided.

[samuelkarp]

Is there a related KEP for improving this, user owned secrets really is the best solution for registry authentication.

+1 to the sentiment that secret management is better outside of containerd directly. We might want to have Kubernetes model the mirror credentials (and go through the KEP process for design), but then we would probably need to also understand whether Kubernetes should manage mirror configuration as well.

[dosubot]

Hi, @markx. I'm Dosu, and I'm helping the containerd team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• The issue involves ImagePullSecrets not working with registry mirrors due to kubelet's credential handling.
• @dmcgowan mentioned that a solution is planned but not included in containerd 2.0, with a proposal in issue [transfer] credential manager plugin #8228.
• @samuelkarp suggested managing secrets outside containerd and possibly involving Kubernetes in modeling mirror credentials.
• You expressed relief that the feature is being considered and that current settings will remain until a new solution is available.

Next Steps:

• Please confirm if this issue is still relevant to the latest version of the containerd repository. If so, you can keep the discussion open by commenting here.
• If there is no further activity, this issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!

[markx]

I saw in release 2.1 that the removal of [plugins."io.containerd.grpc.v1.cri".registry] auths is postponed again, which indicates there's no replacement yet. So this is still relevant.