2.11.0 -> 2.11.1 changed the Registry for Redis and HAProxy from Docker to Amazon ECR breaking Cosign verification

[avo-sepp]

Describe the bug

A recent patch modified the Container Registry used for both Redis and HAProxy. This changed the Registry source from Docker to an Amazon ECR.

Those of us using Cosign to validate the registry source were blocked from upgrading. Because the new Redis version includes an authentication change, it prevented the new Argo from interfacing with Redis. Effectively bringing ArgoCD down and requiring a reinstall. We use ArgoCD to sync ArgoCD, but because ArgoCD went down it had to be manually installed again with kustomize.

Why was the registry changed from Docker to Amazon ECR? Can this be included in the change log more clearly, since it is a breaking change for Cosign users?

To Reproduce

Cosign is enabled and performing image registry validation for Redis and HAProxy

Expected behavior

Upgrading from 2.11.0 -> 2.11.1 should upgrade cleanly.

Screenshots

Version

argocd: v2.11.0+d3f33c0
BuildDate: 2024-05-07T18:31:19Z
GitCommit: d3f33c0
GitTreeState: clean
GoVersion: go1.22.2
Compiler: gc
Platform: darwin/arm64
argocd-server: v2.11.1+9f40df0
BuildDate: 2024-05-21T13:55:56Z
GitCommit: 9f40df0
GitTreeState: clean
GoVersion: go1.21.9
Compiler: gc
Platform: linux/amd64
Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
Helm Version: v3.14.4+g81c902a
Kubectl Version: v0.26.11
Jsonnet Version: v0.20.0

[avo-sepp]

https://github.com/argoproj/argo-cd/blob/8dff209cba4044ec8d0ff8ec32dd79bded7bafaf/manifests/ha/install.yaml#L22943

Line where change occurred. There are a few lines for HAProxy and Redis in this file that were all changed. You can search ecr.aws to find the changes.

f1a449e#diff-f57b731949fe998635a3f1de62d2cd7c5ae7139f7b288af17ee7f7166f3f5b6a

Commit which made the change.

[avo-sepp]

Also worth noting that the non-ha installs still use Docker Hub for HAProxy and Redis. This change only affects HA installers. Sort of a weird thing I noticed. Why use two registry sources?

[crenshaw-dev]

2.11.1 updated the upstream redis-ha chart version. That change included a change in image repo. https://github.com/DandyDeveloper/charts/pull/214/files

I don't think that change should have been included in our change. I'll open a PR to revert to the old image.

[avo-sepp]

Thank you Michael! That makes a lot of sense.

[crenshaw-dev]

@avo-sepp could you check #18329 and let me know if the language of the third paragraph makes sense?

[avo-sepp]

Looks good. I would suggest using the word Policy for Cosign. Something like "make sure your Image Validation policy includes the AWS ECR as an approved registry"

[crenshaw-dev]

Pushed with that additional language! lmk if you'd like me to add you as a co-author, I'd just need an email to include in the commit message. :-)

[avo-sepp]

Sure! I can send you an email in Slack on the CNCF Slack. Do you usually use your Github hide-my-email address for these? That's what I use in commits.