Fix WWW-Authenticate challenge handling with docker registries.

[hpidcock]

Docker v2 registries employ a standard process for authentication by returning challenge in the WWW-Authenticate header. This change attempts to use this process for all generic repositories.

QA steps

This has been tested with quay.io, jfrog artifactory, google cloud artifact repository and digital ocean registry.

• export DOCKER_USERNAME=my-registry.com/juju-test
• make seed-repository
• JUJU_BUILD_NUMBER=0 make push-release-operator-image
• juju bootstrap minikube --config caas-image-repo='{"repository":"my-registry.com/juju-test","username":"<username>","password":"<password>"}'

Documentation changes

N/A

Links

Launchpad bug: https://bugs.launchpad.net/juju/+bug/2039727

Jira card: JUJU-4820

[gustavosr98]

I was missing uploading the jujud-operator (for the juju controller) OCI image with the patch
I used user + password for the registry credentials on the bootstrap
However, I ended with the same result when deploying a workload

# juju deploy postgresql-k8s --channel 14/stable --resource postgresql-image=10.46.90.139/jujusolutions/charmed-postgresql:14.9-22.04_edge

# juju status
Model  Controller  Cloud/Region        Version  SLA          Timestamp
test   mk8s        microk8s/localhost  2.9.46   unsupported  13:13:00-05:00

App             Version  Status   Scale  Charm           Channel    Rev  Address        Exposed  Message
postgresql-k8s           waiting    0/1  postgresql-k8s  14/stable  158  10.152.183.95  no       installing agent

Unit              Workload  Agent  Address      Ports  Message
postgresql-k8s/0  error     lost   10.1.181.24         unknown container reason "ImagePullBackOff": Back-off pulling image "10.46.90.139/jujusolutions/charm-base:ubuntu-22.04"

# microk8s.kubectl describe pod postgresql-k8s-0 -n test

# [..]
Events:
  Type     Reason                           Age              From               Message
  ----     ------                           ----             ----               -------
  Normal   Scheduled                        7s               default-scheduler  Successfully assigned test/postgresql-k8s-0 to black
  Normal   Pulled                           7s               kubelet            Container image "10.46.90.139/jujusolutions/jujud-operator:2.9.46" already present on machine
  Normal   Created                          7s               kubelet            Created container charm-init
  Normal   Started                          6s               kubelet            Started container charm-init
  Normal   Pulling                          6s               kubelet            Pulling image "10.46.90.139/jujusolutions/charm-base:ubuntu-22.04"
  Warning  Failed                           6s               kubelet            Failed to pull image "10.46.90.139/jujusolutions/charm-base:ubuntu-22.04": failed to pull and unpack image "10.46.90.139/jujusolutions/charm-base:ubuntu-22.04": failed to resolve reference "10.46.90.139/jujusolutions/charm-base:ubuntu-22.04": failed to authorize: failed to fetch anonymous token: unexpected status: 401
  Warning  Failed                           6s               kubelet            Error: ErrImagePull
  Normal   Pulling                          6s               kubelet            Pulling image "10.46.90.139/jujusolutions/charmed-postgresql:14.9-22.04_edge"
  Warning  Failed                           6s               kubelet            Failed to pull image "10.46.90.139/jujusolutions/charmed-postgresql:14.9-22.04_edge": failed to pull and unpack image "10.46.90.139/jujusolutions/charmed-postgresql:14.9-22.04_edge": failed to resolve reference "10.46.90.139/jujusolutions/charmed-postgresql:14.9-22.04_edge": failed to authorize: failed to fetch anonymous token: unexpected status: 401
  Warning  Failed                           6s               kubelet            Error: ErrImagePull
  Warning  FailedToRetrieveImagePullSecret  5s (x3 over 7s)  kubelet            Unable to retrieve some image pull secrets (juju-image-pull-secret); attempting to pull the image may not succeed.
  Normal   BackOff                          5s               kubelet            Back-off pulling image "10.46.90.139/jujusolutions/charm-base:ubuntu-22.04"
  Warning  Failed                           5s               kubelet            Error: ImagePullBackOff
  Normal   BackOff                          5s               kubelet            Back-off pulling image "10.46.90.139/jujusolutions/charmed-postgresql:14.9-22.04_edge"
  Warning  Failed                           5s               kubelet            Error: ImagePullBackOff
  
# [..]
Events:
  Type     Reason                           Age                    From     Message
  ----     ------                           ----                   ----     -------
  Warning  FailedToRetrieveImagePullSecret  2m27s (x327 over 72m)  kubelet  Unable to retrieve some image pull secrets (juju-image-pull-secret); attempting to pull the image may not succeed.

[gustavosr98]

Tested again bootstrap now with an online (not local) private repository of Jfrog
It seems working

The local repo from before might have kept the official 2.9.46 jujud-operator image rather than the one provided

# ./juju bootstrap microk8s mk8s --debug --config caas-image-repo='{"repository":"gustavosanchez.jfrog.io/jujusolutions", "serveraddress":"gustavosanchez.jfrog.io/artifactory/api/docker/jujusolutions/v2", "username":"admin", "password":"<PASS>" }'
# [..]

# ./juju deploy postgresql-k8s --channel 14/stable --resource postgresql-image=gustavosanchez.jfrog.io/jujusolutions/charmed-postgresql:14.9-22.04_edge

# [..]
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m52s                default-scheduler  Successfully assigned test/postgresql-k8s-0 to black
  Normal   Pulled     2m51s                kubelet            Container image "gustavosanchez.jfrog.io/jujusolutions/jujud-operator:2.9.46" already present on machine
  Normal   Created    2m51s                kubelet            Created container charm-init
  Normal   Started    2m51s                kubelet            Started container charm-init
  Normal   Pulling    2m51s                kubelet            Pulling image "gustavosanchez.jfrog.io/jujusolutions/charm-base:ubuntu-22.04"
  Normal   Pulled     2m40s                kubelet            Successfully pulled image "gustavosanchez.jfrog.io/jujusolutions/charm-base:ubuntu-22.04" in 10.992s (10.992s including waiting)
  Normal   Created    2m40s                kubelet            Created container charm
  Normal   Started    2m39s                kubelet            Started container charm
  Normal   Pulling    2m39s                kubelet            Pulling image "gustavosanchez.jfrog.io/jujusolutions/charmed-postgresql:14.9-22.04_edge"
  Normal   Pulled     2m14s                kubelet            Successfully pulled image "gustavosanchez.jfrog.io/jujusolutions/charmed-postgresql:14.9-22.04_edge" in 25.728s (25.728s including waiting)
  Normal   Created    2m14s                kubelet            Created container postgresql
  Normal   Started    2m14s                kubelet            Started container postgresql
  Warning  Unhealthy  67s (x14 over 2m7s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 502

[hpidcock]

/merge