Cannot deploy multiple helm dependencies from same private repo server : repository name already exists #7075
Comments
alexmt
added
the
regression
|
I'm also seeing the same thing for deploying dependency charts from multiple public repos with the same host. e.g. for Kyverno plus its policy-reporter: Gives an error log from ArgoCD when syncing: Should be easy enough to work around (just deploying the two dependency charts in two separate Argo apps), but non ideal. |
|
Fixed by #7096 |
|
I am on the newest version (2.3.3) of ArgoCD and I experience more or less the same problem. We have a (test) cluster where I deployed a helm repo secret, and then there's a single reference to the private repository within a helm-chart dependency. Until a few days ago everything worked fine and then this issue appeared: I then deleted the argocd-repo-server pod, and everything seemed to be fine again, however an hour later, the same issue appeared again, this time even effecting the build of other helm-based argocd apps, even though they don't reference the private helm repository. here are the relevant argocd-repo-server logs. It seems as if the helm-repo-add command is executed multiple times, maybe once for each (helm) app (even though most of them don't use it?), it also seems as if from time to time the helm-repo add command succeeds. I am not sure what is the difference between the failing and the succeeding calls Update:I found another way to at least work around the issue:
While I played around with that, I noticed that within the repo server the directory layout looked like this: is it ok that the folder is called |
|
@alexmt can you reopen this issue, or should I create a new one with the description posted above? The problem arises every few hours for me, and is then fixed by the workaround I posted, until it fails again. |
|
@stieglma can you try to narrow down the variables? Sounds like the potentially-relevant variables are:
Maybe you could clone the source repo, point a few apps at it, and try to reproduce the problem. Then push a commit and see if you can get one of the apps to generate manifests successfully. I'm not at all sure, but I suspect that maybe multiple apps are sharing the same build directory on repo-server, and Helm is doing stuff that causes a conflict. Maybe if more than one app weren't generating manifests at the same time, things would work. |
|
If it helps you to understand the issue better I am happy to schedule a video/screenshare call somewhere next week, but for now a little more context here: Our environment is a cluster which is completely managed by ArgoCD (it also manages itself). Overall we have:
"Admin" GitOps RepoProject Gitops RepoThe Chart.yaml referencing the helm chart in the private repo looks like this: apiVersion: v2
name: ikor-generic-chart
version: 1.0.2
dependencies:
- name: generic-chart
version: "1.0.0"
# same url as configured in the helm repo secret
repository: "https://our-repo-url"
alias: test-serviceThe values.yaml for the charts looks like that (shortened): test-service:
deployment:
image: nginx:1.21.6
containerPort: 80 |
|
@crenshaw-dev I tried a couple more things and this is what I found now:
next step will be to add all other helm-chart repos we are using as argocd-secrets and give them a name (I assume this will make them also appear outside the https:/ folder but instead in a folder with the name as foldername), to see if that changes anything |
What is the exit code of that command? |
|
the exit code is 0 |
|
Yep, makes sense. I'm pretty heavily booked for a while, but I've added this to Thursday's contributor office hours doc. Maybe someone there will have an idea. I think where we want to get is a minimal reproducible example. But isolating the relevant variables might require adding logging, dropping into a debugger, etc. If no one's come up with anything by the end of the week, I'd be interested in hopping on a call next week. |
|
I did now create argocd repository secrets for all helm repositories we are using (I just omitted username/password where they are not necessary), and then I assigned this repos to the necessary ArgoCD projects. So far everything seems to be working fine with that change (it was about 6 hours ago, the errors usually appeared much faster). I'll keep an eye on it the next few days. If this actually resolved the problem then I'm sort of fine with it, but it would of course still be much nicer if I didn't need to create an argocd repository secret if it is not actually necessary due to passwords/usernames. unfortunately I won't have time to also attend that meeting, I have something in parallel |
|
Interesting... I have no idea why that would work. What led you to try that? |
|
I thought it might be an interference between "implicit" helm repositories which are just used, but never declared in argocd as a helm repo directly (only as depencies in the Chart.yaml), and the explictly declared helm repo for our private/authorized chart server. With that change I also got rid of that strange Up to now, still everything looking fine |
|
That is fascinating. And you still don't have any Apps pointing directly at those Helm repos? |
|
no, only transitively via dependencies |
|
Wow. Excellent find... I wonder if there's an initial fetch when the repo is added (to validate that the contents contain a chart maybe?) which caches the dependency in a way that allows the At any rate, now we have a workaround and a pretty good hint how to reproduce the issue. |
crenshaw-dev
added
the
workaround
|
Assigning Alex because you has a pretty good idea what was going on here, but I'll also keep this issue in mind. Can't prioritize reproducing it just yet but hopefully soon. |
|
short update, the workaround still works fine, I had no further issues since I created helm-repository secrets for all non-private helm repos as well |
|
@stieglma could you elaborate on this workaround? I'm seeing this too when deploying any app that uses one of our charts as a subchart. We do have that repo added as a repo to argo and have several projects using it, here's some specifics: Example apiVersion: v2
name: mw-example-app
type: application
version: 2.0.0
maintainers:
dependencies:
- name: common
version: ">=0.0.2"
repository: https://our.internal.artifactory.instance/artifactory/helmHere's the repository's secret ( apiVersion: v1
data:
name: artifactory
password: redacted
type: helm
url: https://our.internal.artifactory.instance/artifactory/helm
username: redacted
kind: Secret
metadata:
annotations:
managed-by: argocd.argoproj.io
labels:
argocd.argoproj.io/secret-type: repository
name: repo-872087197
namespace: argocd
type: OpaqueWe also have a set of repositories configured in - name: bitnami
type: helm
url: https://charts.bitnami.com/bitnami
- name: jupyterhub
type: helm
url: https://jupyterhub.github.io/helm-chart/
- name: influxdata
type: helm
url: https://helm.influxdata.com/
- name: grafana
type: helm
url: https://grafana.github.io/helm-charts
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: mw-devtest
namespace: argocd
spec:
clusterResourceWhitelist:
...
description: Middleware (Development/Test)
destinations:
...
orphanedResources:
warn: false
roles:
- jwtTokens:
...
sourceRepos:
- https://our.internal.github/*
- git@our.internal.github/*
- '*'I'm experiencing two problems. If, on my application, I do not enable Sometime after that, this directory fills up with repositories (including the fun I'm on ArgoCD 2.3.3, what am I missing from the workaround? Edit: Is it possible that there's one app with |
|
I don't have any specific settings done for |
|
I was getting this error with |
|
I am getting this error with |
|
any update on this? going to try the workaround above but would be nice if I didn't have to. |
|
still happened in argo-cd-5.4.5 |
If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.
Checklist:
argocd version.Describe the bug
Argocd 2.1.0 fails at deploying helm chart with dependencies fetched from same private help repository server. It triggers the error :
It appears from code that the each new repo using
repo-credswill be named as the host. So, if several repositories use the same host, the code tries to add a helm repo with an existing name.This is a regression from 2.0.5 as this chart was deployed correctly with
repository.credentialsfromargocd-cmTo Reproduce
I have a chart defined as following :
Both dependencies comes from the same private server, here harbor, ie chartmuseum.
I have configured the
repo-credssecret.By parsing the Chart.yaml, the code will add a helm repo named
registry.xxx.ioforregistry.xxx.io/chartrepo/enterprise. Then. for the second depedency, it will try to add the reporegistry.xxx.io/chartrepo/cloudwhich will also be namedregistry.xxx.ioargo-cd/reposerver/repository/repository.go
Line 493 in a894d4b
Helm therefore will fire the error
Error: repository name (registry.xxx.io) already existsExpected behavior
This helm chart was correctly deployed with 2.0.5. It should still be deployed with 2.1.0.
Screenshots
n/a
Version
{ "Version": "v2.1.0+d0b2d55", "BuildDate": "2021-08-20T05:30:54Z", "GitCommit": "d0b2d55e3fb7fe8b17385d6687886de41651f31b", "GitTreeState": "clean", "GoVersion": "go1.16.5", "Compiler": "gc", "Platform": "linux/amd64", "KsonnetVersion": "v0.13.1", "KustomizeVersion": "v4.2.0 2021-06-30T22:49:26Z", "HelmVersion": "v3.6.0+g7f2df64", "KubectlVersion": "v0.21.0", "JsonnetVersion": "v0.17.0" }Logs
The text was updated successfully, but these errors were encountered: