Add support for using ECR as pull-through image cache

[rsafonseca]

This PR introduces a simple way to enable using ECR as a Pull-through image cache, without having to mutate images on the cluster using tools like Kyverno.

Containerd already has support for specifying registry mirrors in kops, but since ECR uses short lived tokens, it's not trivial (or even impossible without adding a few extra hacks on top of it) to use it as a pull-through cache.

This PR also bumps the ecr-credential-provider binary, which before version 1.29.0 specifically tried to parse an ECR repo URL from the image passed, leading to not being possible to enable this feature. This is now resolved in the latest versions.

This PR uses a flag to enable the feature when needed, and adds any server addresses configured in the mirrors to be allowed on the CredentialProviderConfig object that kops configures on the kubelet.

To configure this:

• Create your pull through cache rule in Amazon ECR
• Set spec.containerd.useECRCredentialsForMirrors to true
• Configure your mirrors in spec.containerd.registryMirrors

example:

spec:
  containerd:
    useECRCredentialsForMirrors: true
    registryMirrors:
      docker.io:
        - https://<MY-AWS-ACCOUNT>.dkr.ecr.us-east-1.amazonaws.com/v2/<MY cache rule namespace, e.g docker-hub>

[k8s-ci-robot]

Hi @rsafonseca. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[hakman]

/ok-to-test

[rsafonseca]

Any idea why these e2e tests might be failing @hakman?
It seems odd that they never come up and the log dump references not finding an unrelated role that should belong to another job
e.g.
pull-kops-e2e-cni-amazonvpc:

0530 19:12:47.346328   14816 dumplogs.go:51] /home/prow/go/src/k8s.io/kops/.build/dist/linux/amd64/kops toolbox dump --name e2e-pr16593.pull-kops-e2e-cni-amazonvpc.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /tmp/kops/e2e-pr16593.pull-kops-e2e-cni-amazonvpc.test-cncf-aws.k8s.io/id_ed25519 --ssh-user ubuntu
W0530 19:12:59.466784   50124 aws.go:2055] could not find role "masters.e2e-pr16593.pull-kops-e2e-cni-cilium-ipv6.test-cn-40qra0". Resource may already have been deleted: operation error IAM: GetInstanceProfile, https response error StatusCode: 404, RequestID: 013035b7-7eba-43a9-98dd-abf842145433, NoSuchEntity: Instance Profile masters.e2e-pr16593.pull-kops-e2e-cni-cilium-ipv6.test-cn-40qra0 cannot be found.
W0530 19:13:02.485316   50124 aws.go:2055] could not find role "nodes.e2e-pr16593.pull-kops-e2e-cni-cilium-ipv6.test-cncf-47o1f8". Resource may already have been deleted: operation error IAM: GetInstanceProfile, https response error StatusCode: 404, RequestID: 4bc7f2c4-9c38-4723-85bd-b493cf0794c8, NoSuchEntity: Instance Profile nodes.e2e-pr16593.pull-kops-e2e-cni-cilium-ipv6.test-cncf-47o1f8 cannot be found.

could something be leaking somewhere? or do you think there's any change here that could be somehow related to this weird behaviour?

EDIT: nvm, found the problem... hidden tabs in string literal.. damn vscode 😂
the above behaviour is still odd though, seems like the logs are leaking across jobs when they fail

[hakman]

Please ignore the IPv6 tests

[rsafonseca]

All good then, though it looks like pull-kops-e2e-cni-cilium-eni is flaky, passed on retest.
Any thoughts on this PR @hakman ? Does it make sense to you to add the config option under containerd on the CRD? :)

[sczelo]

Any update about this PR? Do you need any help or test, to be merged faster?
I built the images locally from this repo and tested it on AWS. It's worked for me.

[rsafonseca]

Did you get a chance to review this yet @hakman @johngmyers ? :)

[rsafonseca]

/retest-required

[justinsb]

In office hours, we talked about whether UseECRCredentialsForMirrors should be per mirror or whether we only apply this to urls that "look like" ECR. The (compelling) argument that @rsafonseca made is that people aren't likely to have multiple mirrors so it's not necessary.

👍

[rsafonseca]

@hakman I've removed the for loop from the inline string literal as you had suggested in the office hours, and in fact i removed the whole string literal. Overall the number of lines increased, but it should be more readable and easier to manipulate in the future. 2852bcc

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[itskingori]

/remove-lifecycle stale

[rsafonseca]

rebased again, the number of files that needed their hashes updated inside tests/integration increased dramatically since the last rebase so the bot just moved this from L to XXL 🫠

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from hakman. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@rsafonseca: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kops-e2e-aws-upgrade-k130-ko130-to-klatest-kolatest-many-addons	8b06aa4	link	false	/test pull-kops-e2e-aws-upgrade-k130-ko130-to-klatest-kolatest-many-addons
pull-kops-kubernetes-e2e-ubuntu-gce-build	8b06aa4	link	false	/test pull-kops-kubernetes-e2e-ubuntu-gce-build
pull-kops-e2e-cni-kuberouter	8b06aa4	link	false	/test pull-kops-e2e-cni-kuberouter
pull-kops-e2e-cni-flannel	a783928	link	true	/test pull-kops-e2e-cni-flannel
pull-kops-e2e-gce-cni-kindnet	a783928	link	true	/test pull-kops-e2e-gce-cni-kindnet
pull-kops-e2e-k8s-aws-amazonvpc-u2404	a783928	link	true	/test pull-kops-e2e-k8s-aws-amazonvpc-u2404
pull-kops-e2e-cni-amazonvpc	575d0fa	link	true	/test pull-kops-e2e-cni-amazonvpc
pull-kops-e2e-gce-cni-calico	575d0fa	link	false	/test pull-kops-e2e-gce-cni-calico

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[hakman]

@rsafonseca Please move these to a separate addECRPullThroughPermissions(p *Policy) function and call it only when the flag is set.

[hakman]

Also, add the new option to this test, to see how the generated output looks like. Other than these, LGTM.

kops/tests/integration/update_cluster/containerd-custom/in-v1alpha2.yaml

Lines 13 to 22 in 93d0bed

	containerd:
	skipInstall: true
	registryMirrors:
	docker.io:
	- https://registry-1.docker.io
	"*":
	- http://HostIP2:Port2
	packages:
	urlAmd64: https://github.com/containerd/containerd/releases/download/v1.3.9/cri-containerd-cni-1.3.9-linux-amd64.tar.gz
	hashAmd64: "0000000000000000000000000000000000000000000000000000000000000000"