ApplicationSet scmProvider generator randomly deletes all resources

[H777K]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I use an ApplicationSet with the scmProvider generator. This ApplicationSet templates an Application Resource which templates multiple AppProject and Application Resources which are located on a specific path in my github repository.
For some reason the ApplicationSet Controller logs "generated 0 applications" which will cause a deletion of all generated AppProjects and Applications. In the next reconcile loop the logs shows "generated 1 applications" although no changes were made.

This behaviour is very inconsistent, it can work for weeks without any problem and sometimes it happens every few days.

I have an second cluster which has the almost the same configuration as the first one. The only difference is the path of the AppProject and Application resources in the github repository. I experienced the same issue on this cluster, but never at the same time as the first one.

To Reproduce

I think it´s difficult to reproduce because the behaviour is inconsistent.
But the steps would be:

1. Create github repository which contains multiple AppProject and Application resources
2. create ApplicationSet with scmProvider generator which references the github repository from step 1
3. Apply the ApplicationSet in the k8s cluster

Expected behavior

I would expect that the log of the ApplicationSet controller always shows "generated 1 applications".

Version

argocd: v2.7.6+00c914a
  BuildDate: 2023-06-20T21:18:20Z
  GitCommit: 00c914a948d9e8ad99be8bd82a368fbdeba12f88
  GitTreeState: clean
  GoVersion: go1.19.10
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.7.1+5e54351.dirty
  BuildDate: 2023-05-02T16:35:40Z
  GitCommit: 5e543518dbdb5384fa61c938ce3e045b4c5be325
  GitTreeState: dirty
  GoVersion: go1.19.6
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.0.1 2023-03-14T01:32:48Z
  Helm Version: v3.11.2+g912ebc1
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.19.1

Logs

time="2023-07-01T09:47:47Z" level=info msg="generated 0 applications" generator="{nil nil nil &SCMProviderGenerator{Github:&SCMProviderGeneratorGithub{Organization:myorg,API:https://mygithubenterpriseurl.com/api/v3,TokenRef:nil,AppSecretName:argocd-github-app,AllBranches:false,},Gitlab:nil,Bitbucket:nil,BitbucketServer:nil,Gitea:nil,AzureDevOps:nil,Filters:[]SCMProviderGeneratorFilter{SCMProviderGeneratorFilter{RepositoryMatch:nil,PathsExist:[],PathsDoNotExist:[],LabelMatch:*argocd-applications-int,BranchMatch:nil,},},CloneProtocol:https,RequeueAfterSeconds:nil,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},} nil nil nil nil nil}"
time="2023-07-01T09:47:47Z" level=info msg="Deleted application" app=argocd-int-repo appSet=argocd-int-applicationset
time="2023-07-01T09:47:47Z" level=info msg="end reconcile" applicationset=argocd/argocd-int-applicationset requeueAfter=30m0s
time="2023-07-01T09:56:32Z" level=info msg="Alloc=16771 TotalAlloc=3107088 Sys=45677 NumGC=2032 Goroutines=342"
time="2023-07-01T10:06:32Z" level=info msg="Alloc=17599 TotalAlloc=3108109 Sys=45677 NumGC=2037 Goroutines=342"
time="2023-07-01T10:16:32Z" level=info msg="Alloc=16764 TotalAlloc=3108338 Sys=45677 NumGC=2042 Goroutines=342"
time="2023-07-01T10:20:05Z" level=info msg="generated 1 applications" generator="{nil nil nil &SCMProviderGenerator{Github:&SCMProviderGeneratorGithub{Organization:myorg,API:https://mygithubenterpriseurl.com/api/v3,TokenRef:nil,AppSecretName:argocd-github-app,AllBranches:false,},Gitlab:nil,Bitbucket:nil,BitbucketServer:nil,Gitea:nil,AzureDevOps:nil,Filters:[]SCMProviderGeneratorFilter{SCMProviderGeneratorFilter{RepositoryMatch:nil,PathsExist:[],PathsDoNotExist:[],LabelMatch:*argocd-applications-int,BranchMatch:nil,},},CloneProtocol:https,RequeueAfterSeconds:nil,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},} nil nil nil nil nil}"
time="2023-07-01T10:20:05Z" level=info msg="unchanged Application" app=argocd-int-repo appSet=argocd-int-applicationset
time="2023-07-01T10:20:05Z" level=info msg="end reconcile" applicationset=argocd/argocd-int-applicationset requeueAfter=30m0s

Applied Applicationset

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: argocd-int-applicationset
  namespace: argocd
spec:
  generators:
  - scmProvider:
      cloneProtocol: https
      filters:
      - labelMatch: argocd-applications-int
      github:
        allBranches: false
        api: https://mygithubenterpriseurl.com/api/v3
        appSecretName: argocd-github-app
        organization: myorg
  template:
    metadata:
      name: '{{ repository }}'
    spec:
      destination:
        namespace: '{{ repository }}'
        server: https://kubernetes.default.svc
      project: argocd-applications-int
      source:
        directory:
          exclude: ""
          include: '{*.yml,*.yaml}'
          recurse: true
        path: argocd/applications/int
        repoURL: '{{ url }}'
        targetRevision: main
      syncPolicy:
        syncOptions:
        - CreateNamespace=true

Generated Application from ApplicationSet

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  name: myrepository
  namespace: argocd
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ApplicationSet
    name: argocd-int-applicationset
    uid: 30c57cce-7560-4e4c-adcf-a8d73273f0f7
spec:
  destination:
    namespace: myrepository
    server: https://kubernetes.default.svc
  project: argocd-applications-int
  source:
    directory:
      include: '{*.yml,*.yaml}'
      jsonnet: {}
      recurse: true
    path: argocd/applications/int
    repoURL: https://mygithubenterpriseurl.com/myorg/myrepository.git
    targetRevision: main
  syncPolicy:
    syncOptions:
    - CreateNamespace=true

[H777K]

update: it seems like the scmProvider can´t handle network issues/timeouts regarding the github enterprise instance
Is there any way or config to counteract this behavior?

[crenshaw-dev]

We rely on the go-github client to tell us whether there was an error getting repos. If that client doesn't return an error when we list repos, we'll honor the response.

It's possible that we should inspect further into the response, e.g. validate a 200 return code.

[InsomniaCoder]

I've recently faced a similar problem while our Github Enterprise was under maintenance.

We saw different errors with different status codes such as 500, 502, 503, but they did not trigger the

"generated 0 applications"

However, we saw this specific error

error listing repos: error listing repositories for xxx: invalid character '\u003c' looking for beginning of value

before applicationsetcontroller decided to "generated 0 applications" and delete the previously generated applications.

I'm trying to reproduce but has not succeeded yet.

[siretart]

possibly additional clues from my homelab:

time="2024-05-12T15:28:21Z" level=error msg="error generating application from params" applicationset=argocd/cluster-resources error="error generating params from git: error retrieving Git files: rpc error: code = Internal desc = unable to resolve git revision : Get \"https://gitea.redacted.io/redacted/argocd-autopilot.git/info/refs?service=git-upload-pack\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" generator="{nil nil &GitGenerator{RepoURL:https://gitea.redacted.io/redacted/argocd-autopilot.git,Directories:[]GitDirectoryGeneratorItem{},Files:[]GitFileGeneratorItem{GitFileGeneratorItem{Path:bootstrap/cluster-resources/*.json,},},Revision:,RequeueAfterSeconds:*20,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:&ApplicationSource{RepoURL:,Path:,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,},Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"

the gitea server is running on rather slow machines, and will occasionally tieout and not produce a response. On those situations, it is not desirable to have argocd delete all applications that are defined there.