bug: vmdistributed creates a lot of ReconcileEvents

[Schueni1]

test with v0.69.0

Operator creates a lot of reconcile events per second. I noticed that it does not do this when vmdistributed is first created by the operator. It only occurs when vmdistributed already exists and the operator gets restarted.

kubectl get events

104s        Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
60s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
82s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
29s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
113s        Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
113s        Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
106s        Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
96s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
78s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
68s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
75s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
61s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
93s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
48s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
80s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
91s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
40s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
85s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
87s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
106s        Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
47s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
92s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
77s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
97s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
2m1s        Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
94s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
67s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
101s        Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
60s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
102s        Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
96s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
2m          Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
51s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
57s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
100s        Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
106s        Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
42s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
67s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
81s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
95s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
44s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
88s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
66s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
62s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
36s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
100s        Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
65s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
72s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
102s        Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   starting object update
33s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully
96s         Normal    ReconcileEvent      vmdistributed/vm-victoria-metrics-k8s-stack   reconcile of object finished successfully

[AndrewChubatiuk]

please share you configuration to reproduce, cannot reproduce it with a minimal vmauth config for unauthorized access
also does operator restart frequently in your case?

[Schueni1]

I also noticed, that this does not happen with operator v0.68.4.
Sorry the config is a lot. I can also repost with only the vmauth config.

No the operator does not restart very frequently, but it happens sometimes like when a node goes down.

operator config:

        strategy:
          type: RollingUpdate
          rollingUpdate:
            maxSurge: 0
            maxUnavailable: 1
        crds:
          plain: true
        serviceMonitor:
          enabled: true
          vm: true
          extraLabels:
            victoria-metrics: infrastructure
        fullnameOverride: vm-operator
        replicaCount: 2
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: app.kubernetes.io/name
                    operator: In
                    values:
                    - victoria-metrics-operator
                topologyKey: topology.kubernetes.io/zone
        env:
        - name: VM_PROMETHEUSCONVERTERADDARGOCDIGNOREANNOTATIONS
          value: true
        - name: VM_FILTERPROMETHEUSCONVERTERLABELPREFIXES
          value: "prometheus"
        operator:
          disable_prometheus_converter: false
          enable_converter_ownership: true
        admissionWebhooks:
          enabled: true

k8s-stack vmdistributed config:

        vmdistributed:
          enabled: true
          spec:
            vmauth:
              name: vmauth
              spec:
                rollingUpdate:
                  maxSurge: 0
                  maxUnavailable: 1
                extraArgs:
                  memory.allowedPercent: "95"
                unauthorizedUserAccessSpec:
                  targetRefs:
                  - name: read
                    load_balancing_policy: least_loaded
                  - name: write
                    load_balancing_policy: least_loaded
                  - load_balancing_policy: least_loaded
                    drop_src_path_prefix_parts: 1
                    paths:
                    - /vmagent-cache/.*
                    - /vmagent-cache
                    retry_status_codes:
                    - 500
                    - 502
                    - 503
                    crd:
                      kind: VMAgent
                      name: ''
                      namespace: ''
                      objects:
                      - name: cache-az-1
                        namespace: infrastructure-victoria-metrics
                      - name: cache-az-2
                        namespace: infrastructure-victoria-metrics
                managedMetadata:
                  labels:
                    victoria-metrics: infrastructure
                affinity:
                replicaCount: 2
                selectAllByDefault: true
                affinity:
                  podAntiAffinity:
                    requiredDuringSchedulingIgnoredDuringExecution:
                      - labelSelector:
                          matchExpressions:
                          - key: app.kubernetes.io/name
                            operator: In
                            values:
                            - vmauth
                        topologyKey: topology.kubernetes.io/zone
            zones:
            - name: "1"
            - name: "2"
            zoneCommon:
              vmagent:
                name: "cache-az-%ZONE%"
                spec:
                  remoteWriteSettings:
                    useMultiTenantMode: true
                  rollingUpdate:
                    maxSurge: 0
                    maxUnavailable: 1
                  nodeSelector:
                    topology.kubernetes.io/zone: '%ZONE%'
                  managedMetadata:
                    labels:
                      victoria-metrics: infrastructure
                  statefulMode: true
                  statefulStorage:
                    volumeClaimTemplate:
                      spec:
                        accessModes: ["ReadWriteOnce"]
                        resources:
                          requests:
                            storage: 40Gi
                  extraArgs:
                    remoteWrite.maxBlockSize: "32MB"
                    promscrape.maxScrapeSize: "64MB"
                    promscrape.cluster.name: "cache-%ZONE%"
                  resources:
                    limits:
                      cpu: "1"
                      memory: 4Gi
                    requests:
                      cpu: "250m"
                      memory: 400Mi
                  affinity:
                    podAntiAffinity:
                      requiredDuringSchedulingIgnoredDuringExecution:
                        - labelSelector:
                            matchExpressions:
                            - key: app.kubernetes.io/instance
                              operator: In
                              values:
                              - cache-az-1
                              - cache-az-2
                          topologyKey: topology.kubernetes.io/zone
              vmcluster:
                name: "vmcluster-az-%ZONE%"
                spec:
                  retentionPeriod: 1
                  replicationFactor: 1
                  managedMetadata:
                    labels:
                      victoria-metrics: infrastructure
                  vmstorage:
                    rollingUpdate:
                      maxSurge: 0
                      maxUnavailable: 1
                    nodeSelector:
                      topology.kubernetes.io/zone: '%ZONE%'
                    affinity:
                      podAntiAffinity:
                        requiredDuringSchedulingIgnoredDuringExecution:
                          - labelSelector:
                              matchExpressions:
                              - key: app.kubernetes.io/name
                                operator: In
                                values:
                                - vmstorage
                            topologyKey: kubernetes.io/hostname
                    replicaCount: 1
                    storageDataPath: "/vm-data"
                    resources:
                      limits:
                        cpu: "1"
                        memory: 8Gi
                      requests:
                        cpu: "600m"
                        memory: 5Gi
                    extraArgs:
                      dedup.minScrapeInterval: 30s
                    storage:
                      volumeClaimTemplate:
                        spec:
                          accessModes: ["ReadWriteOnce"]
                          resources:
                            requests:
                              storage: 50Gi
                  vmselect:
                    rollingUpdate:
                      maxSurge: 0
                      maxUnavailable: 1
                    nodeSelector:
                      topology.kubernetes.io/zone: '%ZONE%'
                    storage:
                      volumeClaimTemplate:
                        spec:
                          accessModes: ["ReadWriteOnce"]
                          resources:
                            requests:
                              storage: 4Gi
                    affinity:
                      podAntiAffinity:
                        requiredDuringSchedulingIgnoredDuringExecution:
                          - labelSelector:
                              matchExpressions:
                              - key: app.kubernetes.io/name
                                operator: In
                                values:
                                - vmselect
                            topologyKey: kubernetes.io/hostname
                    replicaCount: 1
                    resources:
                      limits:
                        cpu: "1"
                        memory: 4Gi
                      requests:
                        cpu: "600m"
                        memory: 600Mi
                    extraArgs:
                      search.logSlowQueryDuration: "15s"
                      search.maxConcurrentRequests: "16"
                      search.maxQueueDuration: "30s"
                  vminsert:
                    rollingUpdate:
                      maxSurge: 0
                      maxUnavailable: 1
                    nodeSelector:
                      topology.kubernetes.io/zone: '%ZONE%'
                    affinity:
                      podAntiAffinity:
                        requiredDuringSchedulingIgnoredDuringExecution:
                          - labelSelector:
                              matchExpressions:
                              - key: app.kubernetes.io/name
                                operator: In
                                values:
                                - vminsert
                            topologyKey: kubernetes.io/hostname
                    replicaCount: 1
                    resources:
                      limits:
                        cpu: "1"
                        memory: 4Gi
                      requests:
                        cpu: "600m"
                        memory: 600Mi
                    extraArgs:
                      maxLabelsPerTimeseries: "100"
                  requestsLoadBalancer:
                    enabled: false

[AndrewChubatiuk]

have you upgraded CRDs?

[Schueni1]

Yes I am using latest k8s-stack 0.75.0 and the latest version of the operator chart. Do I have to do more to upgrade CRDs?

[AndrewChubatiuk]

CRDs are not upgraded automatically by default. Please check this doc

[Schueni1]

Thanks for this hint!
But it looks like they are all up to date. Probably because I deleted vmdistributed and recreated it.
This returns nothing.

helm show crds vm/victoria-metrics-k8s-stack --version 0.75.0 | kubectl diff -f -

[AndrewChubatiuk]

was able to reproduce frequent reconcile behaviour only due to stale CRDs, also operator was complaining about absent properties

[Schueni1]

It seems to work now for me. I recreated vmdistributed again and now it stopped frequently reconciling.
Strange.

[Schueni1]

Never mind. It started reconciling again. It just takes some time to start.

[AndrewChubatiuk]

do you observe anything in operator logs?

[Schueni1]

Just that it is constantly reconciling:

{"level":"info","ts":"2026-04-28T12:54:47Z","logger":"controller.VMDistributedReconciler","msg":"object was successfully reconciled","vmdistributed":"vm-victoria-metrics-k8s-stack","namespace":"infrastructure-victoria-metrics"}
{"level":"info","ts":"2026-04-28T12:54:47Z","logger":"controller.VMDistributedReconciler","msg":"object has changes with previous state, applying changes","vmdistributed":"vm-victoria-metrics-k8s-stack","namespace":"infrastructure-victoria-metrics"}
{"level":"info","ts":"2026-04-28T12:54:47Z","logger":"controller.VMDistributedReconciler","msg":"starting reconciliation","vmdistributed":"vm-victoria-metrics-k8s-stack","namespace":"infrastructure-victoria-metrics","name":"infrastructure-victoria-metrics/vm-victoria-metrics-k8s-stack"}
{"level":"info","ts":"2026-04-28T12:54:47Z","logger":"controller.VMDistributedReconciler","msg":"reconciling VMCluster","vmdistributed":"vm-victoria-metrics-k8s-stack","namespace":"infrastructure-victoria-metrics","item":"1/2","name":{"name":"vmcluster-az-1","namespace":"infrastructure-victoria-metrics"}}
{"level":"info","ts":"2026-04-28T12:54:47Z","logger":"controller.VMDistributedReconciler","msg":"ensuring persistent queues are drained","vmdistributed":"vm-victoria-metrics-k8s-stack","namespace":"infrastructure-victoria-metrics","name":"infrastructure-victoria-metrics/vmcluster-az-1"}
{"level":"info","ts":"2026-04-28T12:54:47Z","logger":"controller.VMDistributedReconciler","msg":"all persistent queues were drained","vmdistributed":"vm-victoria-metrics-k8s-stack","namespace":"infrastructure-victoria-metrics","name":"infrastructure-victoria-metrics/vmcluster-az-1"}
{"level":"info","ts":"2026-04-28T12:54:47Z","logger":"controller.VMDistributedReconciler","msg":"reconciling VMCluster","vmdistributed":"vm-victoria-metrics-k8s-stack","namespace":"infrastructure-victoria-metrics","item":"2/2","name":{"name":"vmcluster-az-2","namespace":"infrastructure-victoria-metrics"}}
{"level":"info","ts":"2026-04-28T12:54:47Z","logger":"controller.VMDistributedReconciler","msg":"ensuring persistent queues are drained","vmdistributed":"vm-victoria-metrics-k8s-stack","namespace":"infrastructure-victoria-metrics","name":"infrastructure-victoria-metrics/vmcluster-az-2"}
{"level":"info","ts":"2026-04-28T12:54:47Z","logger":"controller.VMDistributedReconciler","msg":"all persistent queues were drained","vmdistributed":"vm-victoria-metrics-k8s-stack","namespace":"infrastructure-victoria-metrics","name":"infrastructure-victoria-metrics/vmcluster-az-2"}
{"level":"info","ts":"2026-04-28T12:54:47Z","logger":"controller.VMDistributedReconciler","msg":"reconciliation completed","vmdistributed":"vm-victoria-metrics-k8s-stack","namespace":"infrastructure-victoria-metrics","name":"infrastructure-victoria-metrics/vm-victoria-metrics-k8s-stack"}
{"level":"info","ts":"2026-04-28T12:54:47Z","logger":"controller.VMDistributedReconciler","msg":"object was successfully reconciled","vmdistributed":"vm-victoria-metrics-k8s-stack","namespace":"infrastructure-victoria-metrics"}

[AndrewChubatiuk]

do you use ArgoCD?