Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Voyager exporter sidecar isn't exporting any metrics #270

Closed
julianvmodesto opened this issue Jun 28, 2017 · 5 comments

Comments

Projects
None yet
2 participants
@julianvmodesto
Copy link
Contributor

commented Jun 28, 2017

I've tried port-forwarding the Voyager exporter sidecars so I can access http://localhost:56790/voyager.appscode.com/v1beta1/namespaces/default/ingresses/my-app/metrics, but this endpoint is empty.

I've verified that http://localhost:56789/?stats;csv is not empty and I also see that Prometheus is successfully scraping the target -- it's just empty.

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Jun 29, 2017

can you exec into the the HAProxy container and then use curl to see if the prometheus endpoint i returning data or not?

$ kubectl exec -it <pod-name> -c haproxy bash

# Now inside the HAproxy container (this is a Ubuntu based image)
$ apt-get update
$ apt-get install curl
$ curl -v http://127.0.0.1:56790/voyager.appscode.com/v1beta1/namespaces/default/ingresses/my-app/metrics

Also, check the logs for the sidecar from another terminal.

$ kubectl logs -f <pod-name> -c exporter
@tamalsaha

This comment has been minimized.

Copy link
Member

commented Jun 29, 2017

Also, can you please show the version data for exporter sidecar? I want to make sure that you have the latest build for 3.0.0 . To do that change the PullPolicy to Always for the exporter sidecar in HAProxy deployment and restart it.

From your description, it sounds like 7861d96 bug.

@julianvmodesto

This comment has been minimized.

Copy link
Contributor Author

commented Jun 29, 2017

Looks like the metrics are empty

*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 56790 (#0)
> GET /voyager.appscode.com/v1beta1/namespaces/default/ingresses/my-app/metrics HTTP/1.1
> Host: 127.0.0.1:56790
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 0
< Content-Type: text/plain; version=0.0.4
< Date: Thu, 29 Jun 2017 14:34:37 GMT
<
* Connection #0 to host 127.0.0.1 left intact

Logs on exporter -- no acknowledgement of curl

I0628 22:48:45.938867       1 logs.go:19] FLAG: --address=":56790"
I0628 22:48:45.938960       1 logs.go:19] FLAG: --alsologtostderr="false"
I0628 22:48:45.938976       1 logs.go:19] FLAG: --analytics="true"
I0628 22:48:45.938990       1 logs.go:19] FLAG: --haproxy.server-metric-fields="2,3,4,5,6,7,8,9,13,14,15,16,17,18,21,24,33,35,38,39,40,41,42,43,44"
I0628 22:48:45.939004       1 logs.go:19] FLAG: --haproxy.timeout="5s"
I0628 22:48:45.939014       1 logs.go:19] FLAG: --help="false"
I0628 22:48:45.939024       1 logs.go:19] FLAG: --kubeconfig=""
I0628 22:48:45.939039       1 logs.go:19] FLAG: --log.format="\"logger:stderr\""
I0628 22:48:45.939051       1 logs.go:19] FLAG: --log.level="\"info\""
I0628 22:48:45.939063       1 logs.go:19] FLAG: --log_backtrace_at=":0"
I0628 22:48:45.939073       1 logs.go:19] FLAG: --log_dir=""
I0628 22:48:45.939083       1 logs.go:19] FLAG: --logtostderr="true"
I0628 22:48:45.939093       1 logs.go:19] FLAG: --master=""
I0628 22:48:45.939103       1 logs.go:19] FLAG: --stderrthreshold="2"
I0628 22:48:45.939134       1 logs.go:19] FLAG: --v="3"
I0628 22:48:45.939150       1 logs.go:19] FLAG: --vmodule=""
W0628 22:48:47.040246       1 client_config.go:517] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0628 22:48:47.041260       1 exporter.go:57] Starting Voyager exporter...
I0628 22:48:47.041303       1 exporter.go:65] URL pattern: /:apiGroup/v1beta1/namespaces/:namespace/ingresses/:name/metrics
I0628 22:48:47.041332       1 exporter.go:69] Listening on :56790
time="2017-06-28T22:49:18Z" level=info msg="Configuring exporter for appscode ingress my-app in namespace default" source="exporter_handler.go:116"

Voyager version:

kubectl exec -it voyager-my-app-164556864-j43lg -c exporter sh
/ # /voyager version
Version = 3.0.0
VersionStrategy = tag
Os = linux
Arch = amd64
CommitHash = cb43cdcf8f475e4cb7b34685b0226d084ae98753
GitBranch = release-3.0
GitTag = 3.0.0
CommitTimestamp =
BuildTimestamp =
BuildHost =
BuildHostOs =
BuildHostArch =

Tried all of the above after setting imagePullPolicy: Always, but no luck.

@julianvmodesto

This comment has been minimized.

Copy link
Contributor Author

commented Jun 29, 2017

@tamalsaha appreciate the debugging help!

My issue was with the ClusterRole I was using since the exporter needs access to the Ingress.

Here's my new ClusterRole that works.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: voyager
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["voyager.appscode.com"]
  resources: ["ingresses"]
  verbs: ["get"]
@tamalsaha

This comment has been minimized.

Copy link
Member

commented Jun 29, 2017

@julianvmodesto , you will also need get Secret if using secret with stats endpoint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.