Argo-cd fails on service monitor synchronization (and other resources)

[tomaszkiewicz]

Describe the bug

Argo-cd sometimes fails to do synchronization for the whole application. I managed to narrow that to some specific kinds of resources like ServiceMonitor (for Prometheus Operator).
What's more interesting, I can synchronize these resources when I sync them manually one-by-one, but sync for all resources (or subset of resources) fails.

To Reproduce

1. Create app from definition, I prepared a sample repo with podinfo deployment (with some random ServiceMonitor configuration):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: podinfo
  namespace: argocd
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    path: podinfo
    repoURL: https://github.com/tomaszkiewicz/argocd-sample
  syncPolicy:
    automated:
      prune: true

2. Check results in argo-cd UI.

Expected behavior
Sync everything successfully or at least provide a clear reason why sync has failed.

Version
v1.0.2

Logs

argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:22Z" level=info msg="Refreshing app status (normal refresh requested)" application=podinfo
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller W0704 20:01:22.935620       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:22Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=podinfo
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:22Z" level=info msg="Refreshing app status (controller refresh requested)" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller W0704 20:01:22.939448       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:22Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Skipping auto-sync: application status is Synced" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Update successful" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Reconciliation completed" application=cluster full=true time_ms=1148.3231329999999
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Refreshing app status (controller refresh requested)" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller W0704 20:01:24.155238       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Initiated automated sync to 'd099efa289650ef31b4c8b056126d727409cc77c'" application=podinfo reason=OperationStarted type=Normal
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="updated 'podinfo' operation (phase: Running)"
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:d099efa289650ef31b4c8b056126d727409cc77c,Prune:true,DryRun:false,SyncStrategy:nil,Resources:[],Source:nil,}}" application=podinfo
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=podinfo
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Initiated automated sync to 'd099efa289650ef31b4c8b056126d727409cc77c'" application=podinfo
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Updated sync status: Synced -> OutOfSync" application=podinfo reason=ResourceUpdated type=Normal
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Updated health status: Healthy -> Missing" application=podinfo reason=ResourceUpdated type=Normal
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller W0704 20:01:24.183229       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Update successful" application=podinfo
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Reconciliation completed" application=podinfo full=true time_ms=1261.3315329999998
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Refreshing app status (normal refresh requested)" application=podinfo
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller W0704 20:01:24.197004       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=podinfo
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Applying resource Namespace/podinfo in cluster: https://100.64.0.1:443, namespace: default"
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="[kubectl --kubeconfig /dev/shm/164878929 -f - apply -n default --dry-run]"
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller W0704 20:01:24.288950       1 reflector.go:270] github.com/argoproj/argo-cd/util/settings/settings.go:278: watch of *v1.ConfigMap ended with: too old resource version: 6847894 (6848836)
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Skipping auto-sync: another operation is in progress" application=podinfo
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Updated sync status: Synced -> OutOfSync" application=podinfo reason=ResourceUpdated type=Normal
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Updated health status: Healthy -> Missing" application=podinfo reason=ResourceUpdated type=Normal
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Update successful" application=podinfo
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:24Z" level=info msg="Reconciliation completed" application=podinfo full=true time_ms=420.913924
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:25Z" level=info msg="Skipping auto-sync: application status is Synced" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:25Z" level=info msg="Updated health status: Healthy -> Missing" application=cluster reason=ResourceUpdated type=Normal
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:25Z" level=info msg="Update successful" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:25Z" level=info msg="Reconciliation completed" application=cluster full=true time_ms=1121.877899
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:25Z" level=info msg="Refreshing app status (controller refresh requested)" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller W0704 20:01:25.277176       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:25Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:26Z" level=info msg="Skipping auto-sync: application status is Synced" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:26Z" level=info msg="Update successful" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:26Z" level=info msg="Reconciliation completed" application=cluster full=true time_ms=1090.284606
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:27Z" level=info msg="Applying resource Service/podinfo in cluster: https://100.64.0.1:443, namespace: podinfo"
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:27Z" level=info msg="Applying resource Service/podinfo-prometheus-discovery in cluster: https://100.64.0.1:443, namespace: podinfo"
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:27Z" level=info msg="[kubectl --kubeconfig /dev/shm/015547563 -f - apply -n podinfo --dry-run]"
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:27Z" level=info msg="[kubectl --kubeconfig /dev/shm/997950844 -f - apply -n podinfo --dry-run]"
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:28Z" level=info msg="added resource Service/podinfo status: SyncFailed, message: " application=podinfo
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:28Z" level=info msg="Applying resource Deployment/podinfo in cluster: https://100.64.0.1:443, namespace: podinfo"
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:28Z" level=info msg="[kubectl --kubeconfig /dev/shm/891316238 -f - apply -n podinfo --dry-run]"
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:28Z" level=info msg="Applying resource ServiceMonitor/podinfo in cluster: https://100.64.0.1:443, namespace: podinfo"
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:28Z" level=info msg="[kubectl --kubeconfig /dev/shm/866469909 -f - apply -n podinfo --dry-run]"
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:28Z" level=info msg="Updating operation state. phase: Running -> Failed, message: '' -> 'one or more objects failed to apply (dry run)'" application=podinfo
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:29Z" level=info msg="Refreshing app status (controller refresh requested)" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:29Z" level=info msg="updated 'podinfo' operation (phase: Failed)"
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:29Z" level=info msg="Sync operation to d099efa289650ef31b4c8b056126d727409cc77c failed: one or more objects failed to apply (dry run)" application=podinfo reason=OperationCompleted type=Warning
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller W0704 20:01:29.005839       1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:29Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:30Z" level=info msg="Skipping auto-sync: application status is Synced" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:30Z" level=info msg="Update successful" application=cluster
argocd-application-controller-67d84d5b7c-8wstt argocd-application-controller time="2019-07-04T20:01:30Z" level=info msg="Reconciliation completed" application=cluster full=true time_ms=1192.666544

Have you thought about contributing a fix yourself?

I probably can do a little more troubleshooting but I need some info on how to enable some more verbose output to narrow down the issue.

[peterbosalliandercom]

I logged also a related issue with the same strange behaviour: #1887

[alexec]

When I attempt to create this locally, I get this error:

11:30:02  controller | DEBU[0042] Could not create new object of type monitoring.coreos.com/v1, Kind=ServiceMonitor: no kind "ServiceMonitor" is registered for version "monitoring.coreos.com/v1" in scheme "k8s.io/kubernetes/pkg/kubectl/scheme/scheme.go:28" 

[alexec]

Have you installed the CRD please?

[tomaszkiewicz]

Yes, it's running on the cluster where Prometheus is installed, so all necessary CRDs are in place.
Also, when I invoke selective sync for just one resource it succeeds, so it's not related to CRDs, I think...
I managed to run argo controller in debug logging level, but unfortunately it doesn't print why kubectl failed to apply the resources... The escaped json resource printout is valid and basically the same as yaml manifest.
I'm also able to apply all manifests manually with kubectl without any errors or warnings.

Is there any way to get information on why kubectl failed? Or maybe get the exact manifest of what exactly is executed with kubectl?

[alexec]

we've noted that CRDs can take a few moments to install. Perhaps related.

[jessesuen]

This is fixed in #1999 and is a duplicate of #1940