unsupported protocol scheme when syncing already created apps from OCI

[comtihon]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
We store helm charts in Google's Artifact registry.
Repo is added to argocd via secret with oci enabled, connection status is Successful.
Initial sync (application created) works like a charm - latest version is installed according to the wildcard in the version.
However, neither automatic nor manual sync on already created application works.
Instead I get this error:

Unable to deploy revision: error resolving repo revision: rpc error: code = Unknown desc = Get "europe-west3-docker.pkg.dev/<project name>/<registry name>/staging/index.yaml": unsupported protocol scheme ""

The only workaround I found now is to delete the app, as it will be re-created immediately. But it doesn't work for us, as our apps contain managed certificates. Update via deletion makes it unavailable for 5-15 minutes.

To Reproduce

1. create repo

resource "google_artifact_registry_repository" "helm-registry" {
  location      = var.region
  repository_id = "helm-registry"
  format        = "DOCKER" 
}

2. add repo to argocd

resource "kubernetes_secret" "argocd-helm-repo-secret" {
  metadata {
    name      = "argocd-helm-registry"
    namespace = "argocd"
    labels    = {
      "argocd.argoproj.io/secret-type" : "repository"
    }
  }

  data = {
    enableOCI = "true"
    name : "argocd-helm-registry"
    username : "_json_key_base64"
    password : filebase64("<service acc.json>")
    type : "helm"
    url : "europe-west3-docker.pkg.dev/<project name>/<registry name>/${var.environment}"
  }

  type = "Opaque"
}

3. create application via app of apps sync (or add app manually)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: afp-frontend
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  project: default
  source:
    chart: afp-frontend
    helm:
      passCredentials: true
      values: |
        service:
          type: ClusterIP
          port: 80
    repoURL: europe-west3-docker.pkg.dev/<project name>/<registry name>/staging
    targetRevision: 0.*.*
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

4. sync application manually via UI after it is created

Expected behavior

application is synced

Version

argocd: v2.5.5+fc3eaec
  BuildDate: 2022-12-16T16:10:26Z
  GitCommit: fc3eaec6f498ddbe49a5fa9d215a219191fba02f
  GitTreeState: clean
  GoVersion: go1.18.9
  Compiler: gc
  Platform: linux/amd64

Logs

time="2023-02-10T16:52:32Z" level=info msg="received unary call /application.ApplicationService/Sync" grpc.method=Sync grpc.request.content="name:\"afp-frontend\" revision:\"0.*.*\" dryRun:false prune:false strategy:<hook:<syncStrategyApply:<force:false > > > appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-02-10T16:52:32Z" span.kind=server system=grpc
time="2023-02-10T16:52:32Z" level=warning msg="finished unary call with code FailedPrecondition" error="rpc error: code = FailedPrecondition desc = error resolving repo revision: rpc error: code = Unknown desc = Get \"europe-west3-docker.pkg.dev/<project name>/<registry name>/staging/index.yaml\": unsupported protocol scheme \"\"" grpc.code=FailedPrecondition grpc.method=Sync grpc.service=application.ApplicationService grpc.start_time="2023-02-10T16:52:32Z" grpc.time_ms=18.473 span.kind=server system=grpc

Thank you

[alexef]

argocd: v2.5.5+fc3eaec

can you try upgrading to 2.6.1?

[comtihon]

with 2.6.1 I get an error:

Unable to deploy revision: error resolving repo revision: rpc error: code = Unknown desc = unable to get tags: failed to get tags: unable to decode json: invalid character '<' looking for beginning of value

For both new and already created (with 2.5.5) applications.
I use 0.*.* as versions.

[alexef]

the looking for beginning issue is probably related to: #12392 (comment)

I believe the issue is related to the path split between chart name and repoURL.

Can you try this solution ?

[comtihon]

not sure it is my case. I've added a repo with this settings:
europe-west3-docker.pkg.dev/<project id>/<repo>/<branch>
and in the manifest I also have:
repoURL: europe-west3-docker.pkg.dev/<project id>/<repo>/<branch>
chart: <branch>/name
I will still try it, but I don't think that gcp project id and gcp repo name should go to chart.

[comtihon]

unbelievable, but it works!
Thank you very much.

[alexef]

I will still try it, but I don't think that gcp project id and gcp repo name should go to chart.

I agree with you, we need to improve that.

[igitcode]

What's the proper config here? Do you need the full path in both the repo URL and chart name? i.e. <project id>/<repo>/<branch>

I currently have repo added as us-central1-docker.pkg.dev and application config as repoURL: us-central1-docker.pkg.dev and chart as chart: <project id>/<repo>/<name>

I'm running into same issue.

[igitcode]

@comtihon can you share your solution? I tried these 3 combinations and at best I can get an initial deployment, but keeps failing on new versions.

not working:

repo definition: us-central1-docker.pkg.dev/<project_id>/<repo>
project repo def: us-central1-docker.pkg.dev/<project_id>/<repo>
application repo def: us-central1-docker.pkg.dev/<project_id>/<repo>
application repo chart: <project_id>/<repo>/<chart_name>

works on initial deploy, but not new releases.

repo definition: us-central1-docker.pkg.dev
project repo def: us-central1-docker.pkg.dev
application repo def: us-central1-docker.pkg.dev
application repo chart: <project_id>/<repo>/<chart_name>

works on initial deploy, but not new releases.

project repo def: us-central1-docker.pkg.dev/<project_id>/<repo>
repo definition: us-central1-docker.pkg.dev/<project_id>/<repo>
application repo def: us-central1-docker.pkg.dev/<project_id>/<repo>
application repo chart: <chart_name>

[igitcode]

@alexef do you have the proper solution?

[alexef]

@igitcode I don't. Unless the problem is caused by: #12554, which I still need to fix before merging.