Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to list ServiceMonitor #847

Closed
serathius opened this issue Feb 8, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@serathius
Copy link

commented Feb 8, 2018

Kubernetes Provider: GKE
Kubernetes Version: v1.8.6-gke.0
Voyager Deployment Script: curl -fsSL https://raw.githubusercontent.com/appscode/voyager/5.0.0-rc.11/hack/deploy/voyager.sh | bash -s -- --provider=gke --rbac
Prometheus Operator Version: v0.16.0

Voyager Logs:

I0208 09:10:41.062185       1 logs.go:19] FLAG: --address=":56790"
I0208 09:10:41.062409       1 logs.go:19] FLAG: --alsologtostderr="false"
I0208 09:10:41.062491       1 logs.go:19] FLAG: --analytics="true"
I0208 09:10:41.062583       1 logs.go:19] FLAG: --burst="1000000"
I0208 09:10:41.062669       1 logs.go:19] FLAG: --cloud-config=""
I0208 09:10:41.062751       1 logs.go:19] FLAG: --cloud-provider="gke"
I0208 09:10:41.062832       1 logs.go:19] FLAG: --custom-templates=""
I0208 09:10:41.062920       1 logs.go:19] FLAG: --exporter-sidecar-image="appscode/voyager:5.0.0-rc.11"
I0208 09:10:41.063004       1 logs.go:19] FLAG: --haproxy-image="appscode/haproxy:1.7.9-5.0.0-rc.11"
I0208 09:10:41.063088       1 logs.go:19] FLAG: --haproxy.server-metric-fields="2,3,4,5,6,7,8,9,13,14,15,16,17,18,21,24,33,35,38,39,40,41,42,43,44"
I0208 09:10:41.063180       1 logs.go:19] FLAG: --haproxy.timeout="5s"
I0208 09:10:41.063260       1 logs.go:19] FLAG: --help="false"
I0208 09:10:41.063347       1 logs.go:19] FLAG: --ingress-class="voyager"
I0208 09:10:41.063440       1 logs.go:19] FLAG: --kubeconfig=""
I0208 09:10:41.063532       1 logs.go:19] FLAG: --log.format="\"logger:stderr\""
I0208 09:10:41.063615       1 logs.go:19] FLAG: --log.level="\"info\""
I0208 09:10:41.063695       1 logs.go:19] FLAG: --log_backtrace_at=":0"
I0208 09:10:41.063773       1 logs.go:19] FLAG: --log_dir=""
I0208 09:10:41.064340       1 logs.go:19] FLAG: --logtostderr="true"
I0208 09:10:41.064517       1 logs.go:19] FLAG: --master=""
I0208 09:10:41.064604       1 logs.go:19] FLAG: --operator-service="voyager-operator"
I0208 09:10:41.064692       1 logs.go:19] FLAG: --qps="1e+06"
I0208 09:10:41.064777       1 logs.go:19] FLAG: --rbac="true"
I0208 09:10:41.064847       1 logs.go:19] FLAG: --restrict-to-operator-namespace="false"
I0208 09:10:41.064924       1 logs.go:19] FLAG: --resync-period="5m0s"
I0208 09:10:41.064996       1 logs.go:19] FLAG: --stderrthreshold="2"
I0208 09:10:41.065073       1 logs.go:19] FLAG: --v="3"
I0208 09:10:41.065143       1 logs.go:19] FLAG: --vmodule=""
W0208 09:10:41.277092       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0208 09:10:41.278596       1 run.go:121] Starting Voyager operator...
I0208 09:10:41.278680       1 operator.go:55] Ensuring CRD registration
I0208 09:10:44.379682       1 validator.go:26] Skipping ingress external@default, as it is not handled by Voyager.
I0208 09:10:44.481711       1 run.go:153] URL pattern: /:apiGroup/v1beta1/namespaces/:namespace/ingresses/:name/metrics
I0208 09:10:44.481906       1 run.go:157] Listening on :56790
I0208 09:10:44.556550       1 reflector.go:202] Starting reflector *v1.Service (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.556595       1 reflector.go:240] Listing and watching *v1.Service from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.556550       1 reflector.go:202] Starting reflector *v1.Endpoints (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.556929       1 reflector.go:240] Listing and watching *v1.Endpoints from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557119       1 reflector.go:202] Starting reflector *v1.Node (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557133       1 reflector.go:240] Listing and watching *v1.Node from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557432       1 reflector.go:202] Starting reflector *v1.ConfigMap (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557453       1 reflector.go:240] Listing and watching *v1.ConfigMap from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.556983       1 reflector.go:202] Starting reflector *v1.Namespace (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557873       1 reflector.go:240] Listing and watching *v1.Namespace from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558105       1 reflector.go:202] Starting reflector *v1.Secret (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558122       1 reflector.go:240] Listing and watching *v1.Secret from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558318       1 reflector.go:202] Starting reflector *v1beta1.Ingress (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558419       1 reflector.go:240] Listing and watching *v1beta1.Ingress from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558638       1 reflector.go:202] Starting reflector *v1beta1.Ingress (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558658       1 reflector.go:240] Listing and watching *v1beta1.Ingress from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558872       1 reflector.go:202] Starting reflector *v1beta1.Certificate (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558889       1 reflector.go:240] Listing and watching *v1beta1.Certificate from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.559091       1 reflector.go:202] Starting reflector *v1.ServiceMonitor (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.559113       1 reflector.go:240] Listing and watching *v1.ServiceMonitor from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557802       1 reflector.go:202] Starting reflector *v1beta1.DaemonSet (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.559193       1 reflector.go:240] Listing and watching *v1beta1.DaemonSet from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557945       1 reflector.go:202] Starting reflector *v1beta1.Deployment (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.559321       1 reflector.go:240] Listing and watching *v1beta1.Deployment from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.594146       1 services.go:38] [d160fb60-5f1a-46eb-9b69-53568febd56c] Service guestbook@default added
E0208 09:10:44.608339       1 reflector.go:205] github.com/appscode/voyager/pkg/operator/operator.go:84: Failed to list *v1.ServiceMonitor: unknown (get servicemonitors.monitoring.coreos.com)
I0208 09:10:44.619965       1 ingresses.go:35] [0a50a1c9-517f-417f-8bf3-c602399abdff] /, Kind= external@default added
I0208 09:10:44.620188       1 ingresses.go:43] [0a50a1c9-517f-417f-8bf3-c602399abdff] /, Kind= external@default does not match ingress class
I0208 09:10:44.721969       1 services.go:38] [e634eb3b-adab-4e5e-8259-c71f05c96a55] Service kubernetes@default added
I0208 09:10:44.819468       1 services.go:38] [0d729567-ca42-4c2d-8a31-fb30257ea700] Service redis-master@default added
I0208 09:10:44.841952       1 services.go:38] [54a18968-1391-410b-933d-cdaffb137bb4] Service redis-slave@default added
I0208 09:10:44.854698       1 services.go:38] [1c02d01b-5529-40f6-b13c-5c2942aab8d0] Service voyager-operator@default added
I0208 09:10:44.862423       1 services.go:38] [88867376-a353-4692-b14f-fca4bc6b6256] Service default-http-backend@ingress-nginx added
I0208 09:10:44.872821       1 services.go:38] [e26422a4-bc54-4a9e-bc46-169801fd718f] Service ingress-metrics@ingress-nginx added
I0208 09:10:44.881610       1 services.go:38] [19000b50-e5f7-41ee-91f4-289a656b6e3d] Service ingress-nginx@ingress-nginx added
I0208 09:10:44.890661       1 services.go:38] [e41d7936-3ac7-42c1-a32d-1d2a43d6cd81] Service default-http-backend@kube-system added
I0208 09:10:44.898731       1 services.go:38] [4d3c42be-cd0d-461f-8380-3c931068e2a2] Service heapster@kube-system added
I0208 09:10:44.906057       1 services.go:38] [ac006cc5-b8d5-48a4-9916-bb8754ff0c79] Service kube-dns@kube-system added
I0208 09:10:44.913592       1 services.go:38] [95770477-94c1-4c37-bf11-a65f1d2946b2] Service kubelet@kube-system added
I0208 09:10:44.921449       1 services.go:38] [6548fb58-e4eb-417a-a017-ebb63f1b5307] Service kubernetes-dashboard@kube-system added
I0208 09:10:44.928805       1 services.go:38] [997409c6-d407-4ee9-9d63-faf8e824d0fe] Service alertmanager-main@monitoring added
I0208 09:10:44.938681       1 services.go:38] [de036c50-eaa5-4777-80eb-d4b9747364eb] Service alertmanager-operated@monitoring added
I0208 09:10:44.946436       1 services.go:38] [5aca2abd-7cd6-4eda-9adf-349892429139] Service grafana@monitoring added
I0208 09:10:44.954349       1 services.go:38] [b0ae2ca1-eb2b-4b3e-830b-69ff1752b366] Service kube-state-metrics@monitoring added
I0208 09:10:44.973430       1 services.go:38] [0fd7f082-49c4-4783-af8e-d1945f2ba510] Service node-exporter@monitoring added
I0208 09:10:44.988000       1 services.go:38] [caf6367d-7ba2-41b2-840c-071facc2c929] Service prometheus-k8s@monitoring added
I0208 09:10:45.007686       1 services.go:38] [7b614b2a-4e7d-4ded-943f-b6695775da46] Service prometheus-operated@monitoring added
I0208 09:10:45.023737       1 services.go:38] [2e8525ab-cc52-42ff-87a7-e8369ccf053f] Service prometheus-operator@monitoring added
I0208 09:10:45.037730       1 services.go:38] [f9ecef03-cd22-497d-9272-64e86fd98743] Service voyager-operator@monitoring added
I0208 09:10:46.047431       1 reflector.go:240] Listing and watching *v1.ServiceMonitor from github.com/appscode/voyager/pkg/operator/operator.go:84
E0208 09:10:46.094608       1 reflector.go:205] github.com/appscode/voyager/pkg/operator/operator.go:84: Failed to list *v1.ServiceMonitor: unknown (get servicemonitors.monitoring.coreos.com)
I0208 09:10:47.228669       1 reflector.go:240] Listing and watching *v1.ServiceMonitor from github.com/appscode/voyager/pkg/operator/operator.go:84
E0208 09:10:47.261644       1 reflector.go:205] github.com/appscode/voyager/pkg/operator/operator.go:84: Failed to list *v1.ServiceMonitor: unknown (get servicemonitors.monitoring.coreos.com)
...

Operator gets stuck on trying to list service monitors.

@serathius serathius changed the title Faield to list ServiceMonitor Failed to list ServiceMonitor Feb 8, 2018

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Feb 9, 2018

@serathius , operator will not register watcher for ServiceMonitor if you don't have ServiceMonitor CRD registered in your cluster.
https://github.com/appscode/voyager/blob/release-5.0/pkg/operator/servicemonitors.go#L20

You mentioned you are using coreos operator 0.16.x. So, I suppose you have this CRD. So, the issue is we can missing RBAC in the installer https://github.com/appscode/voyager/blob/release-5.0/hack/deploy/rbac.yaml#L29 . Please change it to

- apiGroups:
  - monitoring.coreos.com
  resources:
  - servicemonitors
  verbs: ["get", "create", "update", "patch", "list", "watch"]

This should fix it.

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Feb 9, 2018

Thanks for reporting the issue !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.