Crossplane does not use K8s credential providers to pull packages

[MisterMX]

What happened?

Since #2108 Crossplane does not use the credential providers for k8s in go-contrainerregistry that would use the identity of the kubelet to fetch packages from a private registry. A current workaround is using packagePullSecrets but that's quite cumbersome because it requires a uploading a token manually (at least for AWS).

The providers were deactivated because some issues that seem resolved now: kubernetes/kubernetes#100686

Would be great to see them reactivated in Crossplane.

How can we reproduce it?

Deploy Crossplane on an EKS cluster and let it fetch packages from a private ECR.

What environment did it happen in?

Crossplane version:

Every version since 1.1

[cdenneen]

@hasheddan I upgraded to v1.6.1 and was curious do I need to have the secret in different format for it to work?

Spec:
  Ignore Crossplane Constraints:  false
  Package:                        private.jfrog.io/docker/ap/eks-configuration-apss:0.0.1
  Package Pull Policy:            IfNotPresent
  Package Pull Secrets:
    Name:                      jfrog-docker
  Revision Activation Policy:  Automatic
  Revision History Limit:      0
  Skip Dependency Resolution:  false
Events:
  Type     Reason         Age   From                                      Message
  ----     ------         ----  ----                                      -------
  Warning  UnpackPackage  5s    packages/configuration.pkg.crossplane.io  cannot unpack package: failed to fetch package digest from remote: GET https://private.jfrog.io/artifactory/api/docker/null/v2/token?scope=repository%!A(MISSING)docker%!F(MISSING)ap%!F(MISSING)eks-configuration-apss%!A(MISSING)pull&service=private.jfrog.io: : Bad credentials
  Warning  UnpackPackage  4s    packages/configuration.pkg.crossplane.io  cannot unpack package: failed to fetch package digest from remote: GET https://private.jfrog.io/artifactory/api/docker/null/v2/token?scope=repository%!A(MISSING)docker%!F(MISSING)ap%!F(MISSING)eks-configuration-apss%!A(MISSING)pull&service=private.jfrog.io: : This request is blocked due to recurrent login failures, please try again in 31 seconds
  Warning  UnpackPackage  2s    packages/configuration.pkg.crossplane.io  cannot unpack package: failed to fetch package digest from remote: GET https://private.jfrog.io/artifactory/api/docker/null/v2/token?scope=repository%!A(MISSING)docker%!F(MISSING)ap%!F(MISSING)eks-configuration-apss%!A(MISSING)pull&service=private.jfrog.io: : This request is blocked due to recurrent login failures, please try again in 29 seconds
❯ k get secret jfrog-docker -n crossplane-system
NAME           TYPE                             DATA   AGE
jfrog-docker   kubernetes.io/dockerconfigjson   1      9m30s

[hasheddan]

@cdenneen was this working prior to upgrade and what version were you coming from? We have a patch coming out due to some issues that this change introduced, but I haven't heard anyone report the behavior you are showing here.

If you want to try out the coming patch and see if that addresses the issue you can install with:

helm install crossplane -n crossplane-system https://releases.crossplane.io/build/release-1.6/v1.6.2-rc.0/charts/crossplane-1.6.2-rc.0.tgz

[hasheddan]

@cdenneen confirmed in Slack that this also wasn't supported in v1.4.1, which was the previous version. I have suggested using explicit credentials rather than the docker config.