Cannot update multiple images when using local container registry such as localhost:5000

[tkat0]

Hi team. I'd like to report that someone else may have the same problem.
I have a local (on-premise) Kubernetes environment on microk8s for testing, where I use argocd-image-updater.
If there is anything I can do to help, please let me know.

Describe the bug

I found that argocd-image-updater could not update multiple images under the following conditions.
argocd-image-updater updates only one image.

• Using Kustomize
• Update targets are multiple local docker registry images
  • Note that image includes :port such as localhost:5000 or 192.168.0.100:32000
  • I'm using microk8s local registry
  • (Same for local cluster such as minikube)

This is an example Application to cause the problem.
Note the argocd-image-updater.argoproj.io/image-list.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: 192.168.0.100:32000/api-server,192.168.0.100:32000/app
    argocd-image-updater.argoproj.io/force-update: 'true'
    argocd-image-updater.argoproj.io/update-strategy: latest
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main
    argocd-image-updater.argoproj.io/write-back-target: kustomization
spec:
  project: myapp
  source:
    repoURL: git@github.com:tkat0/xxx.git
    targetRevision: HEAD
    path: kubernetes/myapp/development
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  destination:
    server: https://kubernetes.default.svc
    namespace: default

To Reproduce

It can be reproduced by unit tests.
I added the test to pkg/argocd/update_test.go to confirm.

func Test_UpdateApplication(t *testing.T) {
	// Based on Test_UpdateApplication/Test_kustomize_w/_multiple_images_w/_different_registry_w/_different_tags
	t.Run("Test kustomize w/ multiple images w/ local registry w/ different tags", func(t *testing.T) {
		mockClientFn := func(endpoint *registry.RegistryEndpoint, username, password string) (registry.RegistryClient, error) {
			regMock := regmock.RegistryClient{}
			regMock.On("NewRepository", mock.Anything).Return(nil)
			regMock.On("Tags", mock.Anything).Return([]string{"1.0.2", "1.0.3"}, nil)
			return &regMock, nil
		}

		argoClient := argomock.ArgoCD{}
		argoClient.On("UpdateSpec", mock.Anything, mock.Anything).Return(nil, nil)

		kubeClient := kube.KubernetesClient{
			Clientset: fake.NewFakeKubeClient(),
		}
		annotations := map[string]string{
			// Cannot set `:>=1.0.1` due to similar another problem: msg="Using version constraint '5000/foobar:>=1.0.1' when looking for a new tag"
			common.ImageUpdaterAnnotation:                     "foobar=localhost:5000/foobar:>=1.0.1,foobar=localhost:5000/barbar",
			common.ApplicationWideForceUpdateOptionAnnotation: "true",
		}
		appImages := &ApplicationImages{
			Application: v1alpha1.Application{
				ObjectMeta: v1.ObjectMeta{
					Name:        "guestbook",
					Namespace:   "guestbook",
					Annotations: annotations,
				},
				Spec: v1alpha1.ApplicationSpec{
					Source: v1alpha1.ApplicationSource{
						Kustomize: &v1alpha1.ApplicationSourceKustomize{
							// (A) Commenting this out does the same thing.
							Images: v1alpha1.KustomizeImages{
								"localhost:5000/foobar:1.0.1",
								"localhost:5000/barbar:1.0.1",
							},
						},
					},
				},
				Status: v1alpha1.ApplicationStatus{
					SourceType: v1alpha1.ApplicationSourceTypeKustomize,
					Summary: v1alpha1.ApplicationSummary{
						// (B) Commenting this out does the same thing.
						Images: []string{
							"localhost:5000/foobar:1.0.1",
							"localhost:5000/barbar:1.0.1",
						},
					},
				},
			},
			Images: *parseImageList(annotations),
		}
		res := UpdateApplication(&UpdateConfiguration{
			NewRegFN:   mockClientFn,
			ArgoClient: &argoClient,
			KubeClient: &kubeClient,
			UpdateApp:  appImages,
			DryRun:     false,
		}, NewSyncIterationState())
		assert.Equal(t, 0, res.NumErrors)
		assert.Equal(t, 0, res.NumSkipped)
		assert.Equal(t, 1, res.NumApplicationsProcessed)
		assert.Equal(t, 2, res.NumImagesConsidered)
		assert.Equal(t, 2, res.NumImagesUpdated)
		assert.Equal(t, v1alpha1.KustomizeImages(v1alpha1.KustomizeImages{"localhost:5000/foobar:1.0.3", "localhost:5000/barbar:1.0.3"}), appImages.Application.Spec.Source.Kustomize.Images)
	})
}

You can see that the log is fine. However, the updated image is only "localhost:5000/barbar:1.0.3".

$ go test ./... -v -run Test_UpdateApplication/Test_kustomize_w/_multiple_images_w/_local_registry_w/_different_tags

=== RUN   Test_UpdateApplication/Test_kustomize_w/_multiple_images_w/_local_registry_w/_different_tags
time="2023-01-07T07:21:13+09:00" level=debug msg="Considering this image for update" alias=foobar application=guestbook image_name=foobar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="setting rate limit to 20 requests per second" prefix="localhost:5000" registry="https://localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="Inferred registry from prefix localhost:5000 to use API https://localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="Using no version constraint when looking for a new tag" alias=foobar application=guestbook image_name=foobar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="found 2 from 2 tags eligible for consideration" image="localhost:5000/foobar:1.0.1"
time="2023-01-07T07:21:13+09:00" level=info msg="Setting new image to localhost:5000/foobar:1.0.3" alias=foobar application=guestbook image_name=foobar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=info msg="Successfully updated image 'localhost:5000/foobar:1.0.1' to 'localhost:5000/foobar:1.0.3', but pending spec update (dry run=false)" alias=foobar application=guestbook image_name=foobar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="Considering this image for update" alias=foobar application=guestbook image_name=barbar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="Using no version constraint when looking for a new tag" alias=foobar application=guestbook image_name=barbar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="found 2 from 2 tags eligible for consideration" image="localhost:5000/barbar:1.0.1"
time="2023-01-07T07:21:13+09:00" level=info msg="Setting new image to localhost:5000/barbar:1.0.3" alias=foobar application=guestbook image_name=barbar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=info msg="Successfully updated image 'localhost:5000/barbar:1.0.1' to 'localhost:5000/barbar:1.0.3', but pending spec update (dry run=false)" alias=foobar application=guestbook image_name=barbar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="Using commit message: "
time="2023-01-07T07:21:13+09:00" level=info msg="Committing 2 parameter update(s) for application guestbook" application=guestbook
time="2023-01-07T07:21:13+09:00" level=info msg="Successfully updated the live application spec" application=guestbook
    update_test.go:95: 
                Error Trace:    update_test.go:95
                Error:          Not equal: 
                                expected: v1alpha1.KustomizeImages{"localhost:5000/foobar:1.0.3", "localhost:5000/barbar:1.0.3"}
                                actual  : v1alpha1.KustomizeImages{"localhost:5000/barbar:1.0.3", "localhost:5000/barbar:1.0.3"}

Expected behavior

argocd-image-updater updates all images

Additional context

The cause is due to the implementation of argocd.

When comparing Kustomize image names, argocd checks the image name up to first : as an identifier, so it incorrectly recognizes 192.168.0.100:32000/api-server and 192.168.0.100:32000/app as the same image (192.168.0.100).

• func SetKustomizeImage is called for each image
• func (*ApplicationSourceKustomize) MergeImage and func (KustomizeImage) Match overwrote the same image rather than added.

Therefore, you can't use registry names containing colon :. You can use these workarounds.

• Option 1: Publish Local Registry with a URI that doesn't include a port, such as 192.168.0.101.
  • Use LoadBalancer of service type or Ingress
• Option 2: Use GCR, ECR, or other registries.
  • https://argocd-image-updater.readthedocs.io/en/stable/configuration/registries/

Version

• argocd: v2.5.5+fc3eaec
• argocd-image-updater: v0.12.1

Logs

See To Reproduce" section.

[bkcsfi]

I believe I am also having this issue using gitlab as a registry. I have one argocd project that deploys 2 Applications, one to 'qa' and the other to 'prod'. Each of these have the same image, except the tag is 'develop' in the former case and 'main' in the later.

unfortunately argocd-image-updater only updates the deployed applications when the digest changes on the 'main' tag. And at that point, it actually updates both qa and prod to the same image.

When a new image is built for the develop tag, argocd-image-updater skips it because its only looking at 'main'.

annotations:
    argocd-image-updater.argoproj.io/write-back-method: "git"
    argocd-image-updater.argoproj.io/write-back-target: "kustomization"
    argocd-image-updater.argoproj.io/image-list: "manifest-server-qa=image: mygitlab.company.com:5001/d2d/manifestserver2/bundle:develop, manifest-server=mygitlab.company.com:5001/d2d/manifestserver2/bundle:main"
    argocd-image-updater.argoproj.io/manifest-server-qa.update-strategy: "digest"
    argocd-image-updater.argoproj.io/manifest-server-qa.pull-secret: "pullsecret:argocd/gitlab-registry-credentials"
    argocd-image-updater.argoproj.io/manifest-server.update-strategy: "digest"
    argocd-image-updater.argoproj.io/manifest-server.pull-secret: "pullsecret:argocd/gitlab-registry-credentials"

[jason-rutherford]

When a new image is built for the develop tag, argocd-image-updater skips it because its only looking at 'main'.

I think we are seeing this same issue. If you swap the order of the images in image-list does the behavior swap?