packagePullSecrets not working on v1.6.2

[fahedouch]

What happened?

401 authorization required on private registry when using the latest stable version v1.6.2

it works with v1.6.1

How can we reproduce it?

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: test
spec:
  package: "eu.gcr.io/xxx"
  packagePullPolicy: Always
  packagePullSecrets:
    - name: gcr-json-key

What environment did it happen in?

Crossplane version: v1.6.2

[hasheddan]

@fahedouch thank you for raising this! I am investigating 👍🏻

[hasheddan]

@fahedouch I tested out pulling a private package from gcr using a JSON key file, which appears to be the flow you used here and was able to verify that with the package pull secret it was successful and without it was not. Can you verify that the Secret with name gcr-json-key in the crossplane-system namespace is indeed valid for pull the image in question?

[fahedouch]

@hasheddan , thank you for checking this. did you use the v1.6.2 ?

still have the same error with v1.6.2 :

Events:
  Type     Reason         Age              From                                 Message
  ----     ------         ----             ----                                 -------
  Warning  UnpackPackage  3s (x3 over 4s)  packages/provider.pkg.crossplane.io  cannot unpack package: failed to fetch package digest from remote: HEAD https://eu.gcr.io/xxxxx/manifests/v0.2.0-110.g588be75: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)

but it is ok with v1.6.1

I am using .dockerconfigjson json secret to fetch image from private registry

[hasheddan]

@fahedouch hmm yes I did try with v1.6.2. What is the GCR entry in your dockerconfig using for credential resolution for eu.gcr.io?

[fahedouch]

{
   "auths":{
      "https://eu.gcr.io":{
         "auth":"Base64(_json_key:private_key_sa_gcp)"
      }
   }
}

@hasheddan

[hasheddan]

@fahedouch my apologies, I have repeated the steps and now am able to reproduce. I believe we should be falling through to the pull secret auth since the others in the chain should not give you anything more than anonymous creds. However, I haven't been able to determine the exact behavior yet. I'll make sure to keep you updated as I work on getting a fix in place 👍🏻

[hasheddan]

@fahedouch following up here -- was able to do some experiments this morning and it may be the case that because the default and gcloud resolvers come earlier in the chain, one is getting resolved first, but with credentials that are not valid for the image being pulled. Because it resolves, we never get to the package pull secrets. Do you have any additional information about the environment you are running Crossplane in?

[mcbenjemaa]

My team is blocked, we are trying to move from DTR to artifact-registry.

[hasheddan]

@mcbenjemaa I believe this may have been fixed in google/go-containerregistry#1298 -- I will coordinate with you to test an update in your environment 👍🏻