Error's creating crd's

[ghost]

Hi,

customresourcedefinition.apiextensions.k8s.io/vmclusters.operator.victoriametrics.com configured
customresourcedefinition.apiextensions.k8s.io/vmpodscrapes.operator.victoriametrics.com configured
customresourcedefinition.apiextensions.k8s.io/vmrules.operator.victoriametrics.com configured
customresourcedefinition.apiextensions.k8s.io/vmservicescrapes.operator.victoriametrics.com configured
Error from server (Invalid): error when creating "release/crds/crd.yaml": CustomResourceDefinition.apiextensions.k8s.io "vmagents.operator.victoriametrics.com" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property]
Error from server (Invalid): error when creating "release/crds/crd.yaml": CustomResourceDefinition.apiextensions.k8s.io "vmalertmanagers.operator.victoriametrics.com" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property]
Error from server (Invalid): error when creating "release/crds/crd.yaml": CustomResourceDefinition.apiextensions.k8s.io "vmalerts.operator.victoriametrics.com" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property]
Error from server (Invalid): error when creating "release/crds/crd.yaml": CustomResourceDefinition.apiextensions.k8s.io "vmsingles.operator.victoriametrics.com" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property]

[f41gh7]

Hello,

Can you try to install latest release? v0.2.0. It seems like you're using old version and kubernetes 1.18

You can download new release with bundle:

https://github.com/VictoriaMetrics/operator/releases/download/v0.2.0/bundle_crd.zip

Or use kustomize from quick-start docs:

https://github.com/VictoriaMetrics/operator/blob/master/docs/quick-start.MD#installing-by-kustomize

Soon we'll also introduce helm chart.

[ghost]

Hello,

Can you try to install latest release? v0.2.0. It seems like you're using old version and kubernetes 1.18

You can download new release with bundle:

https://github.com/VictoriaMetrics/operator/releases/download/v0.2.0/bundle_crd.zip

Or use kustomize from quick-start docs:

https://github.com/VictoriaMetrics/operator/blob/master/docs/quick-start.MD#installing-by-kustomize

Soon we'll also introduce helm chart.

Hi,

Thank you very much for response. I managed to install operator, thank you. However, when I am trying to create any of objects, kubectl shows that those are created but nothing in the cluster is appeared though.

Operator logs show this:

{"level":"info","ts":1598641409.6035647,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"ServiceMonitor"}

{"level":"info","ts":1598641411.5120413,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

{"level":"info","ts":1598641411.5619593,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"PodMonitor"}

E0828 19:03:31.973124       1 reflector.go:178] k8s.io/client-go@v12.0.0+incompatible/tools/cache/reflector.go:125: Failed to list *v1beta1.VMSingle: vmsingles.operator.victoriametrics.com is forbidden: User "system:serviceaccount:default:vm-operator" cannot list resource "vmsingles" in API group "operator.victoriametrics.com" at the cluster scope

{"level":"info","ts":1598641413.3622386,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"PrometheusRule"}

I0828 19:03:37.810913       1 request.go:621] Throttling request took 1.746435512s, request: GET:https://10.96.0.1:443/apis/crd.projectcalico.org/v1?timeout=32s

{"level":"info","ts":1598641421.2623627,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"PodMonitor"}

{"level":"info","ts":1598641423.1625116,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"PrometheusRule"}

{"level":"info","ts":1598641423.3122737,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"ServiceMonitor"}

{"level":"info","ts":1598641423.3621266,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

I0828 19:03:47.860806       1 request.go:621] Throttling request took 1.7965649s, request: GET:https://10.96.0.1:443/apis/apiregistration.k8s.io/v1?timeout=32s

{"level":"info","ts":1598641430.0119116,"logger":"controller_vmcluster","msg":"api resource is ready","group":"monitoring.coreos.com/v1","kind":"PrometheusRule"}

{"level":"info","ts":1598641430.8618503,"logger":"controller_vmcluster","msg":"api resource is ready","group":"monitoring.coreos.com/v1","kind":"PodMonitor"}

{"level":"info","ts":1598641432.8621001,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

{"level":"info","ts":1598641433.5622036,"logger":"controller_vmcluster","msg":"api resource is ready","group":"monitoring.coreos.com/v1","kind":"ServiceMonitor"}

{"level":"info","ts":1598641436.8117738,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

{"level":"info","ts":1598641446.8120668,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

{"level":"info","ts":1598641456.8122337,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

E0828 19:04:17.689528       1 reflector.go:178] k8s.io/client-go@v12.0.0+incompatible/tools/cache/reflector.go:125: Failed to list *v1beta1.VMSingle: vmsingles.operator.victoriametrics.com is forbidden: User "system:serviceaccount:default:vm-operator" cannot list resource "vmsingles" in API group "operator.victoriametrics.com" at the cluster scope

{"level":"info","ts":1598641466.8120322,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

{"level":"info","ts":1598641476.8121195,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

{"level":"info","ts":1598641486.812308,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

{"level":"info","ts":1598641496.8119643,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

{"level":"info","ts":1598641506.8119147,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

{"level":"info","ts":1598641516.8122756,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

E0828 19:05:16.949043       1 reflector.go:178] k8s.io/client-go@v12.0.0+incompatible/tools/cache/reflector.go:125: Failed to list *v1beta1.VMSingle: vmsingles.operator.victoriametrics.com is forbidden: User "system:serviceaccount:default:vm-operator" cannot list resource "vmsingles" in API group "operator.victoriametrics.com" at the cluster scope

{"level":"info","ts":1598641526.812076,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

{"level":"info","ts":1598641536.812125,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

{"level":"info","ts":1598641546.8121254,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

E0828 19:05:49.334440       1 reflector.go:178] k8s.io/client-go@v12.0.0+incompatible/tools/cache/reflector.go:125: Failed to list *v1beta1.VMSingle: vmsingles.operator.victoriametrics.com is forbidden: User "system:serviceaccount:default:vm-operator" cannot list resource "vmsingles" in API group "operator.victoriametrics.com" at the cluster scope

{"level":"info","ts":1598641556.811956,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

{"level":"info","ts":1598641566.8121717,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

{"level":"info","ts":1598641576.8123956,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

More recent log here - http://paste.openstack.org/show/797247/

Nevertheless, I ve added prometheus crd's from here https://github.com/helm/charts/tree/master/stable/prometheus-operator#helm-fails-to-create-crds

[f41gh7]

Thanks for the report, i'v managed to find an issue with rbac. It was fixed at master branch, but it wasn't included at v0.2.0 release. Without properly configured rbac - operator wasn't able to start items reconciliation.

For now you can solve the problem with two options:

• manually edit release/operator/rbac.yaml, you have to replace vmsingle with vmsingles sed -i 's/vmsingle/vmsingles/g' release/operator/rbac.yaml
• apply fresh role.yaml from master branch kubectl apply -f https://raw.githubusercontent.com/VictoriaMetrics/operator/master/config/rbac/role.yaml it already contains fix.

Btw, if you dont have prometheus operator crd, you can disable converter by setting following env variables to the operator deployment

- name: VM_ENABLEDPROMETHEUSCONVERTER_PODMONITOR  
  value: "false"
- name:  VM_ENABLEDPROMETHEUSCONVERTER_SERVICESCRAPE:
  value: "false"
- name:  VM_ENABLEDPROMETHEUSCONVERTER_PROMETHEUSRULE
  value: "false"
- name:  VM_ENABLEDPROMETHEUSCONVERTER_PROBE
  value: "false"

[ghost]

Thanks for the report, i'v managed to find an issue with rbac. It was fixed at master branch, but it wasn't included at v0.2.0 release. Without properly configured rbac - operator wasn't able to start items reconciliation.

For now you can solve the problem with two options:

* manually edit release/operator/rbac.yaml, you have to replace vmsingle with vmsingles `sed -i 's/vmsingle/vmsingles/g' release/operator/rbac.yaml `

* apply fresh role.yaml from master branch ` kubectl apply -f https://raw.githubusercontent.com/VictoriaMetrics/operator/master/config/rbac/role.yaml` it already contains fix.

Btw, if you dont have prometheus operator crd, you can disable converter by setting following env variables to the operator deployment

- name: VM_ENABLEDPROMETHEUSCONVERTER_PODMONITOR  
  value: "false"
- name:  VM_ENABLEDPROMETHEUSCONVERTER_SERVICESCRAPE:
  value: "false"
- name:  VM_ENABLEDPROMETHEUSCONVERTER_PROMETHEUSRULE
  value: "false"
- name:  VM_ENABLEDPROMETHEUSCONVERTER_PROBE
  value: "false"

Thanks a lot, it worked. I have one more question - in quick start docs, everything is showed in conjunction with vmsingle deployment of victoriametrics. However, I am a bit confused how can I use those tools (like vmagent, vmalert, etc) with VMcluster deployment? I ve found some referal to that https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmagent#splitting-data-streams-among-multiple-systems ; but I am still not getting how can you specify remoteWrite to ClusterIP's of VMcluster. I ve tried vminstert but it doesnt seem like the right way http://vminsert-vmcluster.monitoring.svc.cluster.local:8480/api/v1/write

[f41gh7]

Great!
Yep, need to add it to quick start docs for VMCluster with VMAgent example. URL for connection looks different and it a bit confusing. It must be "http://minsert-vmcluster.monitoring.svc.cluster.local:8480/insert/0/prometheus/api/v1/write"

We have separate article for High-Availability setup link.

[mhoppe-cedar]

Hey folks,

I am having a similar experience while trying to get VMAlert up-and-running. My rbac includes the vmsingles update and I have added the ServiceMonitor, PodMonitor, PrometheusRule, and probe CRD's to my manifests. All the errors cleared from my vm-operator logs EXCEPT for probes.

{"level":"info","ts":1600292637.302681,"logger":"setup","msg":"Registering Components."} {"level":"info","ts":1600292638.208253,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1600292638.2095063,"logger":"setup","msg":"starting vmconverter clients"} {"level":"info","ts":1600292638.2096996,"logger":"setup","msg":"vmconverter was started"} {"level":"info","ts":1600292638.2097168,"logger":"setup","msg":"starting manager"} {"level":"info","ts":1600292638.209796,"logger":"controller_vmcluster","msg":"waiting for api resource","group":"monitoring.coreos.com/v1","kind":"probe"} {"level":"info","ts":1600292638.2098212,"logger":"controller_vmcluster","msg":"waiting for api resource","group":"monitoring.coreos.com/v1","kind":"PodMonitor"} {"level":"info","ts":1600292638.2102165,"logger":"controller_vmcluster","msg":"waiting for api resource","group":"monitoring.coreos.com/v1","kind":"ServiceMonitor"} I0916 21:43:58.209797 1 leaderelection.go:242] attempting to acquire leader lease monitoring/57410f0d.victoriametrics.com... {"level":"info","ts":1600292638.2104821,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1600292638.2098868,"logger":"controller_vmcluster","msg":"waiting for api resource","group":"monitoring.coreos.com/v1","kind":"PrometheusRule"} I0916 21:44:09.260154 1 request.go:621] Throttling request took 1.045369885s, request: GET:https://172.20.0.1:443/apis/autoscaling/v2beta2?timeout=32s {"level":"info","ts":1600292653.5619023,"logger":"controller_vmcluster","msg":"api resource is ready","group":"monitoring.coreos.com/v1","kind":"PodMonitor"} I0916 21:44:15.693506 1 leaderelection.go:252] successfully acquired lease monitoring/57410f0d.victoriametrics.com {"level":"info","ts":1600292655.693786,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMPodScrape","controller":"vmpodscrape","source":"kind source: /, Kind="} {"level":"info","ts":1600292655.693868,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMRule","controller":"vmrule","source":"kind source: /, Kind="} {"level":"info","ts":1600292655.6939366,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMServiceScrape","controller":"vmservicescrape","source":"kind source: /, Kind="} {"level":"info","ts":1600292655.694055,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMAgent","controller":"vmagent","source":"kind source: /, Kind="} {"level":"info","ts":1600292655.6937616,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMProbe","controller":"vmprobe","source":"kind source: /, Kind="} {"level":"info","ts":1600292655.6941226,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMAlertmanager","controller":"vmalertmanager","source":"kind source: /, Kind="} {"level":"info","ts":1600292655.694193,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMSingle","controller":"vmsingle","source":"kind source: /, Kind="} {"level":"info","ts":1600292655.6942513,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMCluster","controller":"vmcluster","source":"kind source: /, Kind="} {"level":"info","ts":1600292655.697714,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMAlert","controller":"vmalert","source":"kind source: /, Kind="} {"level":"info","ts":1600292655.7976162,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMPodScrape","controller":"vmpodscrape"} {"level":"info","ts":1600292655.799677,"logger":"controller_vmcluster","msg":"api resource is ready","group":"monitoring.coreos.com/v1","kind":"PrometheusRule"} {"level":"info","ts":1600292655.9977002,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMRule","controller":"vmrule"} {"level":"info","ts":1600292655.9985664,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMServiceScrape","controller":"vmservicescrape"} {"level":"info","ts":1600292656.2006073,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMSingle","controller":"vmsingle","source":"kind source: /, Kind="} {"level":"info","ts":1600292656.201182,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMProbe","controller":"vmprobe"} {"level":"info","ts":1600292656.2977555,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMAgent","controller":"vmagent","source":"kind source: /, Kind="} {"level":"info","ts":1600292656.3010488,"logger":"controller_vmcluster","msg":"api resource is ready","group":"monitoring.coreos.com/v1","kind":"ServiceMonitor"} {"level":"info","ts":1600292656.301604,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"} {"level":"info","ts":1600292656.3980575,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMCluster","controller":"vmcluster","source":"kind source: /, Kind="} {"level":"info","ts":1600292656.4986207,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMAlert","controller":"vmalert","source":"kind source: /, Kind="} {"level":"info","ts":1600292656.6976576,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMAlertmanager","controller":"vmalertmanager","source":"kind source: /, Kind="} {"level":"info","ts":1600292656.69775,"logger":"controller","msg":"Starting workers","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMPodScrape","controller":"vmpodscrape","worker count":1} {"level":"info","ts":1600292656.6978254,"logger":"controller","msg":"Starting workers","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMRule","controller":"vmrule","worker count":1} {"level":"info","ts":1600292656.6979182,"logger":"controllers.VMRule","msg":"Reconciling VMRule","vmrule":"default/test-vmrule"} {"level":"info","ts":1600292656.697943,"logger":"controllers.VMRule","msg":"listing vmalerts","vmrule":"default/test-vmrule"} {"level":"info","ts":1600292656.6979887,"logger":"controllers.VMRule","msg":"current count of vm alerts: ","vmrule":"default/test-vmrule","len":1} {"level":"info","ts":1600292656.697994,"logger":"controllers.VMRule","msg":"updating or creating cm for vmalert","vmrule":"default/test-vmrule"} {"level":"info","ts":1600292656.698,"logger":"controllers.VMRule","msg":"reconciling vmalert rules","vmrule":"default/test-vmrule"} {"level":"info","ts":1600292656.6980593,"logger":"factory","msg":"selected Rules","rules":"","namespace":"default","vmalert":"vmalert-config"} {"level":"info","ts":1600292656.6986854,"logger":"controller","msg":"Starting workers","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMServiceScrape","controller":"vmservicescrape","worker count":1} {"level":"info","ts":1600292656.6987786,"logger":"controllers.VMServiceScrape","msg":"Reconciling VMServiceScrape","vmservicescrape":"default/vmalert-vmalert-config"} {"level":"info","ts":1600292656.6988418,"logger":"controllers.VMServiceScrape","msg":"found vmagent objects ","vmservicescrape":"default/vmalert-vmalert-config","len: ":0} {"level":"info","ts":1600292656.6988478,"logger":"controllers.VMServiceScrape","msg":"reconciled serviceScrape","vmservicescrape":"default/vmalert-vmalert-config"} {"level":"info","ts":1600292656.8987894,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMAlertmanager","controller":"vmalertmanager"} {"level":"info","ts":1600292657.2984,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMAgent","controller":"vmagent"} {"level":"info","ts":1600292657.298621,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMCluster","controller":"vmcluster","source":"kind source: /, Kind="} {"level":"info","ts":1600292657.2987006,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMCluster","controller":"vmcluster"} {"level":"info","ts":1600292657.2988794,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMAlert","controller":"vmalert"} {"level":"info","ts":1600292657.2992747,"logger":"controller","msg":"Starting workers","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMAgent","controller":"vmagent","worker count":1} {"level":"info","ts":1600292657.2993398,"logger":"controller","msg":"Starting workers","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMCluster","controller":"vmcluster","worker count":1} {"level":"info","ts":1600292657.299355,"logger":"controller","msg":"Starting workers","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMAlertmanager","controller":"vmalertmanager","worker count":1} {"level":"info","ts":1600292657.2993824,"logger":"controller","msg":"Starting workers","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMAlert","controller":"vmalert","worker count":1} {"level":"info","ts":1600292657.2994947,"logger":"controllers.VMAlert","msg":"Reconciling","vmalert":"default/vmalert-config"} {"level":"info","ts":1600292657.2995574,"logger":"factory","msg":"selected Rules","rules":"","namespace":"default","vmalert":"vmalert-config"} {"level":"info","ts":1600292657.299612,"logger":"factory","msg":"no Rule changes","reconcile":"rulesCm","vmalert":"vmalert-config","namespace":"default","vmalert":"vmalert-config"} {"level":"info","ts":1600292657.2996314,"logger":"controllers.VMAlert","msg":"found configmaps for vmalert","vmalert":"default/vmalert-config"," len ":1,"map names":["vm-vmalert-config-rulefiles-0"]} {"level":"info","ts":1600292657.3013835,"logger":"controller","msg":"Starting workers","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMProbe","controller":"vmprobe","worker count":1} {"level":"info","ts":1600292657.3016617,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMSingle","controller":"vmsingle"} {"level":"info","ts":1600292657.3983483,"logger":"factory","msg":"no Rule changes","reconcile":"rulesCm","vmalert":"vmalert-config","namespace":"default","vmalert":"vmalert-config"} {"level":"info","ts":1600292657.3984659,"logger":"controllers.VMRule","msg":"created rules maps count","vmrule":"default/test-vmrule","count":1} {"level":"info","ts":1600292657.701903,"logger":"controller","msg":"Starting workers","reconcilerGroup":"operator.victoriametrics.com","reconcilerKind":"VMSingle","controller":"vmsingle","worker count":1} {"level":"info","ts":1600292657.702875,"logger":"factory","msg":"generating new deployment","controller":"vmalert.crud","vmalert":"vmalert-config"} {"level":"info","ts":1600292657.7029204,"logger":"factory","msg":"generating new deployment","controller":"vmalert.crud","vmalert":"vmalert-config"} {"level":"info","ts":1600292657.8012912,"logger":"factory","msg":"reconciled vmalert deploy","controller":"vmalert.crud","vmalert":"vmalert-config"} {"level":"info","ts":1600292657.8013287,"logger":"controllers.VMRule","msg":"reconciled vmalert rules","vmrule":"default/test-vmrule"} {"level":"info","ts":1600292657.801353,"logger":"controllers.VMRule","msg":"alert rule was reconciled","vmrule":"default/test-vmrule"} {"level":"info","ts":1600292657.8018458,"logger":"factory","msg":"reconciled vmalert deploy","controller":"vmalert.crud","vmalert":"vmalert-config"} {"level":"info","ts":1600292658.012678,"logger":"factory","msg":"vmalert svc reconciled","controller":"vmalert.service.crud","vmalert":"vmalert-config"} {"level":"info","ts":1600292658.02776,"logger":"controllers.VMAlert","msg":"vmalert reconciled","vmalert":"default/vmalert-config"} {"level":"info","ts":1600292659.0615668,"logger":"controller_vmcluster","msg":"api resource doesnt exist, waiting for it","group":"monitoring.coreos.com/v1","kind":"probe"}

The last line just repeats forever and I am wondering if this is my blocker to why my alertmanager isn't being notified.

Any guidance is greatly appreciated!

Thanks

[f41gh7]

Hello, @FHIR360 , according to log file, VMAlert deployment was created at namespace: default, with deployment name vmalert-vmalert-config.

You can check it by executing commands:

kubectl -n default get deploy
kubectl -n default get pods

Let me know, if it doesnt help.

[mhoppe-cedar]

Thank you very much, @f41gh7 , definitely a major issue in my implementation!

[tenmozes]

@FHIR360 close an issue, feel free to reopen if any question