Wildcard support in OCI Helm repositories does not work #12392

Danielkem · 2023-02-10T07:43:34Z

Checklist:

I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
I've included steps to reproduce the bug.
I've pasted the output of argocd version.

Describe the bug

I try to create an application from an OCI Helm repository with wildcard (*) as version instead of explicitly specifying the version of the chart which fails with the following error:

Unable to create application: application spec for x is invalid: InvalidSpecError: Unable to generate manifests in : rpc error: code = Unknown desc = unable to get tags: failed to get tags: unable to decode json: invalid character '<' looking for beginning of value

As seen here:

When I instead pin the version it does work as expected:

To Reproduce

The repository is Harbor:

Expected behavior

It should work as described in #10641

Screenshots

Version

argocd-server: v2.6.1+3f143c9
  BuildDate: 2023-02-08T18:51:05Z
  GitCommit: 3f143c9307f99a61bf7049a2b1c7194699a7c21b
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.7 2022-08-02T16:35:54Z
  Helm Version: v3.10.3+g835b733
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.19.1

Logs

Paste any relevant application logs here.

The text was updated successfully, but these errors were encountered:

alexef · 2023-02-10T08:30:25Z

@Danielkem thank you for reporting this. is Harbour a cloud registry or something you run on premise? I will try to reproduce the problem, I just need to know how to set this up.

I know OCI Helm wildcard works at least for AWS ECR and Azure Container Registry.

Danielkem · 2023-02-10T08:34:03Z

Hey @alexef,
We use https://artifacthub.io/packages/helm/bitnami/harbor/15.2.2 to deploy our Harbor instance on AKS. Should be quite easy to setup and reproduce with that helm chart. If there is anything else I can do to help with debugging this please tell me.

ankitabhopatkar13 · 2023-02-10T14:35:54Z

We are facing somewhat similar issue with Gitlab OCI registry for helm charts.

When we try to use "*" in targetRevision to pick the latest helm chart from the OCI registry, we receive the following error -

Unable to save changes: application spec for <app_name> is invalid: InvalidSpecError: Unable to generate manifests in : rpc error: code = Unknown desc = unable to get tags: failed to get tags: failed tags part: invalid response: 401 Unauthorized {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"reposito...

Looking at the code a bit it fails probably here and if we are not wrong, it is implemented considering AWS ECR API specification.

Note that if we use a fixed version in targetRevision it does not fail because it skips the GetTags call.

blakepettersson · 2023-02-10T15:33:21Z

Looking at the code a bit it fails probably here and if we are not wrong, it is implemented considering AWS ECR API specification.

Nope that's implemented considering the OCI spec.

blakepettersson · 2023-02-10T15:48:02Z

@ankitabhopatkar13 does that work when you use the helm CLI? Gitlab has a huge disclaimer about its Helm support not being ready for production use...

ankitabhopatkar13 · 2023-02-10T16:07:08Z

@blakepettersson can you please elaborate on what exactly do you mean with using helm CLI?

We can do a helm show on the private Gitlab Registry after logging in to the registry. The commands that we ran are as follows -

helm registry login -u ...
helm show all oci://registry.gitlab.com/<path>

We can't do a helm search on the repo as helm does not support it for OCI registries.

blakepettersson · 2023-02-10T16:16:59Z

@ankitabhopatkar13 something like

helm install --dry-run oci://registry.gitlab.com/<path> --generate-name --version "0.2.*" | grep helm.sh/chart | head -n1

or

helm pull oci://registry.gitlab.com/<path> --version "0.2.*"

ankitabhopatkar13 · 2023-02-10T16:19:40Z

Yes, we could do -

helm pull oci://registry.gitlab.com/<path> --version "2.1.*"

and even

helm pull oci://registry.gitlab.com/<path> --version "*"

comtihon · 2023-02-13T09:35:58Z

Same issue, when upgrading from 2.5.5 to 2.6.1

ankitabhopatkar13 · 2023-02-13T13:17:12Z

@blakepettersson Just to add more context:

The only way we(cc: @jorianvo) could list all the tags at https://registry.gitlab.com/v2/<path>/tags/list is by first getting a bearer token from Gitlab and then passing that token as a header Authorization: Bearer.

As per this Www-Authenticate in the response header tells us how to authenticate to list all the tags, the header value we get is:

Www-Authenticate: Bearer realm="https://gitlab.com/jwt/auth",service="container_registry",scope="repository:<path>:pull"

Then we generate a bearer token by running the following:

curl --user 'dummy:<personal-access-token>'  'https://gitlab.com/jwt/auth?client_id=docker&offline_token=true&service=container_registry&scope=repository:<path>:pull'

Then we use the token to list all the tags by running the following:

curl --header "Authorization: Bearer <token_from_above>" "https://registry.gitlab.com/v2/<path>/tags/list"

OpsMaya · 2023-02-14T12:12:37Z

Maybe related issue, GHCR and ArgoCD 2.6.1,
got a 404 error while specifying a wildcard

Unable to create application: application spec for test is invalid: InvalidSpecError: Unable to generate manifests in : rpc error: code = Unknown desc = unable to get tags: failed to get tags: failed tags part: invalid response: 404 Not Found {"error":"Not Found"}

But when specifying the correct version its working.

blakepettersson · 2023-02-17T11:05:54Z

@ankitabhopatkar13 that was indeed very helpful, thanks a lot!

From what I can see in the code the GetTags logic relies on HTTP Basic authentication, without performing the Docker registry flow. This is definitely something that needs to be addressed. I don't use Gitlab nor GHCR so I can't be much help here sadly...

bennesp · 2023-02-20T14:21:44Z

Maybe related issue, GHCR and ArgoCD 2.6.1.

I get a 404 trying using this snippet:

chart: metacontroller-helm
repoURL: ghcr.io/metacontroller
targetRevision: v4.7.*

and a rpc error: code = Unknown desc = unable to get tags: failed to get tags: failed tags part: invalid response: 401 Unauthorized {"errors":[{"code":"UNAUTHORIZED","message":"authentication required"}]} trying to use this:

chart: metacontroller/metacontroller-helm
repoURL: ghcr.io
targetRevision: v4.7.*

Instead, pinning the targetRevision: v4.7.8 works fine

alexef · 2023-02-20T15:25:14Z

@ankitabhopatkar13

Then we use the token to list all the tags by running the following

do I understand correctly that for helm template or helm pull, you use --user dummy:<personal-access-token>, but for listing tags you need to exchange that credential pair for a JWT token?

alexef · 2023-02-20T15:49:35Z

@detvdl / @blakepettersson after checking a bit https://github.com/oras-project/oras-go/blob/31c574b137ccfb3e8627e47f138c4a44b9efa8e5/registry/remote/auth/client.go#L156 and also remembering #11708, I don't see an easy way out of this mess.

We need to replace my GetTags implementation with the oras-go one. that should handle various registries with various policies (and hopefully continue to support AWS ECR and Azure CR without issues).

ankitabhopatkar13 · 2023-02-21T07:33:53Z

@alexef

do I understand correctly that for helm template or helm pull, you use --user dummy:, but for listing tags you need to exchange that credential pair for a JWT token?

Yes, that is correct!

alexef · 2023-02-21T08:05:12Z

#12554 should fix it

EladDolev · 2023-03-07T13:22:14Z

Also getting the same with Google Artifact Registry

ComparisonError
rpc error: code = Unknown desc = unable to get tags: failed to get tags: unable to decode json: invalid character '<' looking for beginning of value

ben741 · 2023-04-26T19:06:31Z

Maybe related issue, GHCR and ArgoCD 2.6.1, got a 404 error while specifying a wildcard
Unable to create application: application spec for test is invalid: InvalidSpecError: Unable to generate manifests in : rpc error: code = Unknown desc = unable to get tags: failed to get tags: failed tags part: invalid response: 404 Not Found {"error":"Not Found"}
But when specifying the correct version its working.

I'm seeing the same error message as @OpsMaya reported above, but with Azure CR and ArgoCD 2.6.7.

To clarify, is it likely to be the same root cause as the other reports here (with the mention of json in the error message), or is it a separate issue?

alexef · 2023-05-11T14:36:32Z

@Danielkem / @bennesp / @EladDolev / @ben741 appreciate if you can confirm the fix for Azure/GCR (now in master)

ben741 · 2023-05-12T14:37:54Z

Thanks @alexef, I'm all set up to test this on Azure, but it looks like the latest tag (v2.8.0+d4714f3) is still just before your commit with the fix, so I'm just waiting on that to see if the issue goes away.

ben741 · 2023-05-12T17:57:51Z

I tested with the new image (v2.8.0+6a3a81c.dirty) and Azure CR. Pulling a specific tag like 1.2.3 works fine, but using a wildcard like * or 1.2.* fails with a 401 status:

rpc error: code = Unknown desc = unable to get tags:
failed to get tags: 
GET "https://<examplecr>.azurecr.io/v2/abc/charts/def/tags/list": 
GET "https://<examplecr>.azurecr.io/oauth2/token?scope=repository%3Aabc%2Fcharts%2Fdef%3Ametadata_read%2Cpull&service=examplecr.azurecr.io": response status code 401: 
unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.

ben741 · 2023-05-16T02:41:12Z

After using oras-go directly, I figured out my issue: I've been using a repoURL like mycr.azurecr.io/foo and chart path charts/bar, but on this line oras-go expects a registry (i.e. hostname, mycr.azurecr.io in my example - not the full repo URL). If I change the repoURL to mycr.azurecr.io and the chart path to foo/charts/bar, the tags are fetched successfully and wildcard support seems to work.

@alexef / @crenshaw-dev should I open a new issue for this?

wardenlym · 2023-06-05T09:12:43Z

Maybe related issue, GHCR and ArgoCD 2.6.1, got a 404 error while specifying a wildcard
Unable to create application: application spec for test is invalid: InvalidSpecError: Unable to generate manifests in : rpc error: code = Unknown desc = unable to get tags: failed to get tags: failed tags part: invalid response: 404 Not Found {"error":"Not Found"}
But when specifying the correct version its working.
I'm seeing the same error message as @OpsMaya reported above, but with Azure CR and ArgoCD 2.6.7.

To clarify, is it likely to be the same root cause as the other reports here (with the mention of json in the error message), or is it a separate issue?

I met this same problem when work with Azure CR :

argocd version
argocd: v2.7.3+unknown
  BuildDate: 2023-05-25T00:59:55Z
  GitCommit:
  GitTreeState:
  GitTag: 2.7.3
  GoVersion: go1.20.4
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.7.2+cbee7e6.dirty

Is there anything update about this issue?

alexef · 2023-06-05T10:24:19Z

After using oras-go directly, I figured out my issue: I've been using a repoURL like mycr.azurecr.io/foo and chart path charts/bar, but on this line oras-go expects a registry (i.e. hostname, mycr.azurecr.io in my example - not the full repo URL). If I change the repoURL to mycr.azurecr.io and the chart path to foo/charts/bar, the tags are fetched successfully and wildcard support seems to work.

@ben741 yes, this is a known long standing issue. If there's no other issue opened, please create one.

@wardenlym just to clarify, even with latest master, you still get a 404? what is the registry/repoURL combination do you use?

wardenlym · 2023-06-06T02:59:40Z

@wardenlym just to clarify, even with latest master, you still get a 404? what is the registry/repoURL combination do you use?

The combination I used before was:

  sources:
    - chart: chart-name
      repoURL: myreg.azurecr.io/helm-chart/charts

After seeing your suggestion, I updated to latest master and changed form to:

  sources:
    - chart: helm-chart/charts/chart-name
      repoURL: myreg.azurecr.io

and now it works fine.

mchudinov · 2023-06-14T14:16:16Z

I use ArgoCD version 2.7.4 and there is still this issue with private GitHub container registry. Neither manipulation with chart name - repoUrl name helps. It is still the same error

rpc error: code = Unknown desc = unable to get tags: failed to get tags: failed tags part: invalid response: 401 Unauthorized {"errors":[{"code":"UNAUTHORIZED","message":"authentication required"}]}

My application configuration

kind: Application
apiVersion: argoproj.io/v1alpha1
metadata:
  name: example-helm
  namespace: argocd
spec:
  project: example
  source:
    chart: myorganization/example-helm
    repoURL: ghcr.io
    targetRevision: 0.2.*

Is the fix released? What version of ArgoCD has the fix?

blakepettersson · 2023-06-14T14:22:03Z

@mchudinov, it's not yet released - it's due for 2.8. Try the latest master and see if the problem persists.

crenshaw-dev · 2023-06-14T14:24:59Z

Ah, forgot that. Closing again. 😆

jlec · 2023-08-14T09:02:24Z

Does work for me in 2.8 :/

alexef · 2023-08-14T09:33:31Z

@jlec what is your set up? which Helm repository are you using, how is it configured? You might be running into different issues than the root cause of this ticket.

stefangluszek · 2023-08-30T13:37:13Z

It's still doesn't work with the latest released version of ArgoCD (2.8.2) and Google Artifact Registry. It seems the credentials are not passed correctly when requesting the tag list and the api server returns a 403.

jlec · 2023-08-30T20:09:27Z

@jlec what is your set up? which Helm repository are you using, how is it configured? You might be running into different issues than the root cause of this ticket.

I am using ghcr.io. For example

➜  helm show all oci://ghcr.io/jlec/uptime-kuma
Pulled: ghcr.io/jlec/uptime-kuma:0.1.22
Digest: sha256:a8f900f5567a28dc60d8546145dbbc4e33ef4b02d2a8d9323b9ba7e32a8340a8
apiVersion: v2
appVersion: 1.22.1
description: A Helm chart for Uptime Kuma
name: uptime-kuma
type: application
version: 0.1.22

Asking ArgoCD to install 0.1.* results in

Unable to save changes: application spec for kuma is invalid: InvalidSpecError: Unable to generate manifests in ./: rpc error: code = Unknown desc = unable to get tags: failed to get tags: GET "https://ghcr.io/v2/jlec/uptime-kuma/tags/list": GET "https://ghcr.io/token?scope=repository%3Ajlec%2Fuptime-kuma%3Apull&service=ghcr.io": response status code 401: unauthorized: authentication required

gris-gris · 2023-12-24T04:39:22Z

#16394 (comment)

Danielkem added the bug Something isn't working label Feb 10, 2023

alexef mentioned this issue Feb 13, 2023

unsupported protocol scheme when syncing already created apps from OCI #12406

Closed

3 tasks

alexef self-assigned this Feb 20, 2023

alexef mentioned this issue Feb 21, 2023

feat: use oras-go library to enable more complex OCI Helm authentication #12554

Merged

10 tasks

lhriley mentioned this issue Apr 11, 2023

Unable to use OCI Registry with sub helm charts #12436

Closed

crenshaw-dev closed this as completed in #12554 May 11, 2023

onematchfox mentioned this issue May 20, 2023

Update semver to v3.2.1 argoproj-labs/terraform-provider-argocd#282

Closed

crenshaw-dev reopened this Jun 14, 2023

crenshaw-dev closed this as completed Jun 14, 2023

ngodec mentioned this issue Jul 17, 2023

Could not get tags from Google Artifact Registry argoproj-labs/argocd-image-updater#579

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wildcard support in OCI Helm repositories does not work #12392

Wildcard support in OCI Helm repositories does not work #12392

Danielkem commented Feb 10, 2023

alexef commented Feb 10, 2023

Danielkem commented Feb 10, 2023

ankitabhopatkar13 commented Feb 10, 2023 •

edited

Loading

blakepettersson commented Feb 10, 2023

blakepettersson commented Feb 10, 2023

ankitabhopatkar13 commented Feb 10, 2023

blakepettersson commented Feb 10, 2023

ankitabhopatkar13 commented Feb 10, 2023

comtihon commented Feb 13, 2023

ankitabhopatkar13 commented Feb 13, 2023

OpsMaya commented Feb 14, 2023 •

edited

Loading

blakepettersson commented Feb 17, 2023

bennesp commented Feb 20, 2023 •

edited

Loading

alexef commented Feb 20, 2023 •

edited

Loading

alexef commented Feb 20, 2023

ankitabhopatkar13 commented Feb 21, 2023

alexef commented Feb 21, 2023

EladDolev commented Mar 7, 2023

ben741 commented Apr 26, 2023

alexef commented May 11, 2023

ben741 commented May 12, 2023

ben741 commented May 12, 2023

ben741 commented May 16, 2023

wardenlym commented Jun 5, 2023 •

edited

Loading

alexef commented Jun 5, 2023

wardenlym commented Jun 6, 2023 •

edited

Loading

mchudinov commented Jun 14, 2023 •

edited

Loading

blakepettersson commented Jun 14, 2023

crenshaw-dev commented Jun 14, 2023

jlec commented Aug 14, 2023

alexef commented Aug 14, 2023

stefangluszek commented Aug 30, 2023

jlec commented Aug 30, 2023 •

edited

Loading

gris-gris commented Dec 24, 2023

Wildcard support in OCI Helm repositories does not work #12392

Wildcard support in OCI Helm repositories does not work #12392

Comments

Danielkem commented Feb 10, 2023

alexef commented Feb 10, 2023

Danielkem commented Feb 10, 2023

ankitabhopatkar13 commented Feb 10, 2023 • edited Loading

blakepettersson commented Feb 10, 2023

blakepettersson commented Feb 10, 2023

ankitabhopatkar13 commented Feb 10, 2023

blakepettersson commented Feb 10, 2023

ankitabhopatkar13 commented Feb 10, 2023

comtihon commented Feb 13, 2023

ankitabhopatkar13 commented Feb 13, 2023

OpsMaya commented Feb 14, 2023 • edited Loading

blakepettersson commented Feb 17, 2023

bennesp commented Feb 20, 2023 • edited Loading

alexef commented Feb 20, 2023 • edited Loading

alexef commented Feb 20, 2023

ankitabhopatkar13 commented Feb 21, 2023

alexef commented Feb 21, 2023

EladDolev commented Mar 7, 2023

ben741 commented Apr 26, 2023

alexef commented May 11, 2023

ben741 commented May 12, 2023

ben741 commented May 12, 2023

ben741 commented May 16, 2023

wardenlym commented Jun 5, 2023 • edited Loading

alexef commented Jun 5, 2023

wardenlym commented Jun 6, 2023 • edited Loading

mchudinov commented Jun 14, 2023 • edited Loading

blakepettersson commented Jun 14, 2023

crenshaw-dev commented Jun 14, 2023

jlec commented Aug 14, 2023

alexef commented Aug 14, 2023

stefangluszek commented Aug 30, 2023

jlec commented Aug 30, 2023 • edited Loading

gris-gris commented Dec 24, 2023

ankitabhopatkar13 commented Feb 10, 2023 •

edited

Loading

OpsMaya commented Feb 14, 2023 •

edited

Loading

bennesp commented Feb 20, 2023 •

edited

Loading

alexef commented Feb 20, 2023 •

edited

Loading

wardenlym commented Jun 5, 2023 •

edited

Loading

wardenlym commented Jun 6, 2023 •

edited

Loading

mchudinov commented Jun 14, 2023 •

edited

Loading

jlec commented Aug 30, 2023 •

edited

Loading