Unable to use in-cluster kube client for "test" command

[janpieper]

Describe the bug

Running argocd-image-updater test [...] with credentials in a kubernetes Secret fails when running inside a kubernetes cluster.

To Reproduce

1. Add a custom registry and use a kubernetes Secret for credentials.

registries:
  - name: myfancyregistry
    ping: yes
    api_url: https://myfancyregistry.azurecr.io
    prefix: myfancyregistry.azurecr.io
    credentials: secret:myfancyns/myfancysecret#auth

2. Exec into the kubernetes pod

kubectl exec -it -n argocd-image-updater argocd-image-updater-7ff85b79cd-lmcjs -- /bin/sh

3. Run argocd-image-updater test [...]

$ argocd-image-updater --registries-conf /app/config/registries.conf test myfancyregistry.azurecr.io/example

Expected behavior

Use in-cluster kube client.

Additional context

I think I already tracked down the error. When running argocd-image-updater test [...] inside a kubernetes cluster, I get the following warning, because kubeClient is nil:

argocd-image-updater/pkg/registry/registry.go

Lines 193 to 198 in 71b1dac

	if (credSrc.Type == image.CredentialSourcePullSecret || credSrc.Type == image.CredentialSourceSecret) && kubeClient == nil {
	log.WithContext().
	AddField("registry", ep.RegistryAPI).
	Warnf("cannot user K8s credentials without Kubernetes client")
	return fmt.Errorf("could not fetch image tags")
	}

The kubeClient is nil, because I do not specify --kubeconfig, so it skips creating a kube client:

argocd-image-updater/cmd/main.go

Lines 344 to 349 in 71b1dac

	if kubeConfig != "" {
	kubeClient, err = getKubeConfig(ctx, "", kubeConfig)
	if err != nil {
	log.Fatalf("could not create K8s client: %v", err)
	}
	}

Checking for kubeConfig != "" is wrong because "" (empty string) tells getKubeConfig to return an in-cluster kube client (see line 661):

argocd-image-updater/cmd/main.go

Lines 646 to 662 in 71b1dac

	func getKubeConfig(ctx context.Context, namespace string, kubeConfig string) (*kube.KubernetesClient, error) {
	var fullKubeConfigPath string
	var kubeClient *kube.KubernetesClient
	var err error
	if kubeConfig != "" {
	fullKubeConfigPath, err = filepath.Abs(kubeConfig)
	if err != nil {
	return nil, fmt.Errorf("cannot expand path %s: %v", kubeConfig, err)
	}
	}
	if fullKubeConfigPath != "" {
	log.Debugf("Creating Kubernetes client from %s", fullKubeConfigPath)
	} else {
	log.Debugf("Creating in-cluster Kubernetes client")
	}

I had a look into what argocd-image-updater run [...] does in this case and found out it does not check for kubeConfig != "". Instead if checks for !disableKubernetes:

argocd-image-updater/cmd/main.go

Lines 526 to 535 in 71b1dac

	if !disableKubernetes {
	ctx := context.Background()
	cfg.KubeClient, err = getKubeConfig(ctx, cfg.ArgocdNamespace, kubeConfig)
	if err != nil {
	log.Fatalf("could not create K8s client: %v", err)
	}
	if cfg.ClientOpts.ServerAddr == "" {
	cfg.ClientOpts.ServerAddr = fmt.Sprintf("argocd-server.%s", cfg.KubeClient.Namespace)
	}
	}

I think using !disableKubernetes would be the correct check for argocd-image-updater test [...] too.

Version

0.10.1 (via Helm chart)

Logs

INFO[0000] getting image                                 image_name=example registry=myfancyregistry.azurecr.io
DEBU[0000] rate limit for https://myfancyregistry.azurecr.io is 2147483647 
INFO[0000] Loaded 1 registry configurations from /app/config/registries.conf 
WARN[0000] cannot user K8s credentials without Kubernetes client  registry="https://myfancyregistry.azurecr.io"
FATA[0000] could not set registry credentials: could not fetch image tags

[jannfis]

As opposed to the run command, the test command was intended to be run from your workstation, and not from within a cluster. So that's the reason why the test command expects a kube config for building the actual K8s client.

But I agree that there might be a valid use case for running test from within your cluster, and thus requiring an in-cluster K8s client.

So probably, we should mimic the run command in that behaviour:

• Use Kubernetes client by default for the test command
• If --kubeconfig is given, use that configuration for building the K8s client. Otherwise, build an in-cluser K8s client.
• Similar to run command, provide a --disable-kubernetes option for the test command that disables the creation and usage of a K8s client

[jannfis]

Closed by #293