Add a patch to make ECR credentials work

[jonjohnsonjr]

Fixes #1996

This patches in kubernetes/kubernetes#75585 and kubernetes/kubernetes#75587 in order to use the updated ECR credential provider. We'll be able to drop these patches once we've bumped our k8s dependencies to 1.15, but that's pretty far off.

Proposed Changes

• Add a patch to enable fetching credentials for ECR.

Release Note

[jonjohnsonjr]

/retest

[jonjohnsonjr]

cc @symfrog @Justin2997 either of you interested in testing this out?

[Justin2997]

@jonjohnsonjr That seems good to me. I don't have the infra to test it anymore. Maybe @lavoiedn can test it.

[lavoiedn]

Funny story that... Our cluster got destroyed and we're still bringing it back 😂

Not able to test it at the moment, will update if I get a chance to fix our setup!

[sebgoa]

@tzununbekov can you try to test this

[tzununbekov]

I've installed jonjohnsonjr:k8schain-patch branch on EKS cluster and tried to create simple knservice with image from private ECR - revision has failed to fetch the image with following message:

Unable to fetch image "730809894724.dkr.ecr.us-east-2.amazonaws.com/knative-test:latest": unsupported status code 401; body: Not Authorized

while k8s job on the same cluster and image succeeded:

  ...
  Normal  Pulling    27m   kubelet, ip-192-168-6-223.us-east-2.compute.internal  pulling image "730809894724.dkr.ecr.us-east-2.amazonaws.com/knative-test:latest"
  Normal  Pulled     27m   kubelet, ip-192-168-6-223.us-east-2.compute.internal  Successfully pulled image "730809894724.dkr.ecr.us-east-2.amazonaws.com/knative-test:latest"
  Normal  Created    27m   kubelet, ip-192-168-6-223.us-east-2.compute.internal  Created container

Is that a test case or do I miss something?

[tzununbekov]

UPD: looks like I should run update-dep.sh script to apply "ecr-provider-hack" patch before tests, but it complains that github.com/knative/test-infra doesn't met constraints

[lavoiedn]

Update from us, we got our cluster back and we're just about to need to test this.

@januaryh you'll need this fix when you try to fetch the image for the rabbitmq adapter/controller.

CC: @alexgervais

[jonjohnsonjr]

@tzununbekov is it a fortio issue? knative/test-infra#968

[jonjohnsonjr]

This seems to be working again

[tzununbekov]

I'm still getting Unable to fetch image: unsupported status code 401; body: Not Authorized error in knservice revisions with ECR registry. Can someone else confirm it?

[ryanbrainard]

I just tried this as well, and still getting Unable to fetch image <image id>: unsupported status code 401; body: Not Authorized. I'm really sure I applied the patch correctly, but here's what I did:

• Baseline: Existing EKS cluster already running Knative v0.6.0
• Checked out this PR
• Setup cluster admin (not sure if needed)
• Deploy cert-manager (not sure if needed, but i did see errors about not finding a cert manager before i did this)
• Deploy Knative Serving
• Checked my existing service using a private registry (still broken)
• Created a new service using a private registry (still broken)

Did I miss any steps?

Here is one of the log lines from the controller after I deployed:

{"level":"error","ts":"2019-06-21T07:49:10.808Z","logger":"controller.revision-controller","caller":"revision/revision.go:197","msg":"Failed to reconcile","commit":"6a2d54f","knative.dev/controller":"revision-controller","knative.dev/traceid":"b8b19590-58f3-4017-8e9f-909dd7a85777","knative.dev/key":"default/shaas1-qkqm7","phase":"image digest","error":"unsupported status code 401; body: Not Authorized\n","stacktrace":"github.com/knative/serving/pkg/reconciler/revision.(*Reconciler).reconcile\n\t/Users/rbrainard/Development/go/src/github.com/knative/serving/pkg/reconciler/revision/revision.go:197\ngithub.com/knative/serving/pkg/reconciler/revision.(*Reconciler).Reconcile\n\t/Users/rbrainard/Development/go/src/github.com/knative/serving/pkg/reconciler/revision/revision.go:98\ngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).processNextWorkItem\n\t/Users/rbrainard/Development/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:330\ngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).Run.func1\n\t/Users/rbrainard/Development/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:282"}

cc: @mattmoor

[jonjohnsonjr]

@tzununbekov @ryanbrainard thanks both for trying this -- can you see if this brings up anything relevant?

$ kubectl logs -n knative-serving $(kubectl get pod -n knative-serving -l app=controller -oname) | grep AWS

E.g. I hit this but that's expected because I'm running on GKE:

aws_credentials.go:76] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.

[ryanbrainard]

@jonjohnsonjr Seeing this:

$ kubectl logs -n knative-serving $(kubectl get pod -n knative-serving -l app=controller -oname) | grep AWS

ERROR: logging before flag.Parse: E0621 07:43:40.372529       1 aws_credentials.go:76] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.

[ryanbrainard]

@jonjohnsonjr Good news, I was able to get this to work! It turned out to be a kiam issue that was blocking this from working last time I tried it. I had to give ECR access via IAM to the knative-serving/controller pod and then it all worked.

@tzununbekov perhaps you have a similar issue? I was able to debug this by looking at the controller's events (kubectl describe pods -n knative-serving -l app=controller) and then tracing those back to the AWS request ids in CloudTrail (or just look for recent AssumeRole events with failures). It all made sense once I found the problem, but the indirection that kiam adds tripped me up a bit, so the CloudTrail logs were helpful there.

[ryanbrainard]

@jonjohnsonjr I just re-tested this again after your rebase on a brand new fresh cluster (I was mucking around a lot on the other one, so wanted to be 100% sure). Still works 👍

[jonjohnsonjr]

/assign @mattmoor

[jonjohnsonjr]

/retest

[mattmoor]

/lgtm
/approve

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jonjohnsonjr, mattmoor

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mattmoor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment