Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support direct scrapping via Prometheus #593

Closed
jalberto opened this issue Oct 6, 2017 · 10 comments

Comments

Projects
None yet
3 participants
@jalberto
Copy link

commented Oct 6, 2017

helm chart stable/prometheus support annotations for configuration, so I will like to add annotations to operator like this:

prometheus.io/scrape: "true"

@jalberto jalberto changed the title support annotations in exporter/stats service support annotations in operator Oct 6, 2017

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Oct 7, 2017

I have looked into this. There are 2 ways to get Prometheus ready metrics in Voyager. We expose this using https://github.com/appscode/haproxy_exporter .

  • side-car-exporter for HAProxy pods: Currently voyager can automatically create ServiceMonitor objects for prometheus-operator. https://github.com/appscode/voyager/blob/master/docs/user-guide/ingress/stats-and-prometheus.md#using-prometheus . In this mode, a side-car container is run with HAProxy pods.

  • out-of-pod exporter: Voyager operator itselff can expose metrics for all HAProxy pods it manages using its own built-in exporter service. This is exposed via port 56790 port. This can't be used by prometheus-operator due to coreos/prometheus-operator#387 . But if someone if configuring prometheus directly, this could used to scrape HAProxy metrics. The limitation of this approach is that, prometheus.io/scrape: "true" annotations can't be applied to the operator service pod, if you have more than one Ingress to manager.

To allow direct scraping of HAProxy stats, 2 options are available:

  • opt 1: we only create the stats service for each HAProxy, then apply the annotations like prometheus.io/scrape: "true" but use the exporter built-in voyager exporter. This can be generalized by supporting ingress.appscode.com/annotations-stats: {"k": "v"} annotation that allows use to apply any annotation to the Stats service for HAproxy pods. Potential issues:

    • if there are many HAProxy pods, and only one operator pod, I wonder if that becomes a bottle neck. I don't this to be a major issue.
  • opt 2: In this mode, we create the stats service and add side-car but instead of creating a ServiceMonitor, we apply the prometheus.io/scrape: "true" annotations on stats service to scrape annotations from side-car. This can be enabled by annotation like ingress.appscode.com/monitoring-agent: prometheus-direct or something.

My question what makes sense, folks?

cc: @julianvmodesto

@jalberto

This comment has been minimized.

Copy link
Author

commented Oct 7, 2017

Great analysis!

IMHO I prefer the approach that creates less objects (so less moving parts). Anyway if exporter is enabled is because we want to use prometheus, so prometheus annotations can be auto-applied.

@julianvmodesto

This comment has been minimized.

Copy link
Contributor

commented Oct 8, 2017

I'm happy with the existing behavior with a sidecar HAProxy exporter and a ServiceMonitor since we run several voyager pods, with about one voyager pod per Node in our cluster.

@tamalsaha tamalsaha changed the title support annotations in operator Support direct scrapping via Prometheus Oct 12, 2017

@tamalsaha tamalsaha added this to the 4.1.0 milestone Oct 12, 2017

tamalsaha added a commit that referenced this issue Nov 14, 2017

tamalsaha added a commit that referenced this issue Nov 14, 2017

tamalsaha added a commit that referenced this issue Nov 14, 2017

tamalsaha added a commit that referenced this issue Nov 14, 2017

tamalsaha added a commit that referenced this issue Nov 14, 2017

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Nov 14, 2017

#703 adds supports a new value for annotation
ingress.appscode.com/monitoring-agent: prometheus-builtin

If this annotation is used, the stats-server will be annotated with prometheus.io/ annotations

apiVersion: voyager.appscode.com/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  namespace: demo
  annotations:
    ingress.appscode.com/stats: 'true'
    ingress.appscode.com/monitoring-agent: 'prometheus-builtin'
spec:
  backend:
    serviceName: test-server
    servicePort: 80
  rules:
  - host: voyager.demo
    http:
      paths:
      - backend:
          serviceName: test-server
          servicePort: 80



$ kubectl get svc -n demo voyager-test-ingress-stats -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    ingress.appscode.com/origin-api-schema: voyager.appscode.com/v1beta1
    ingress.appscode.com/origin-name: test-ingress
    prometheus.io/path: /voyager.appscode.com/v1beta1/namespaces/demo/ingresses/test-ingress/metrics
    prometheus.io/port: "56790"
    prometheus.io/scrape: "true"
  creationTimestamp: 2017-11-14T23:38:32Z
  labels:
    feature: stats
    origin: voyager
    origin-api-group: voyager.appscode.com
    origin-name: test-ingress
  name: voyager-test-ingress-stats
  namespace: demo
  resourceVersion: "3189"
  selfLink: /api/v1/namespaces/demo/services/voyager-test-ingress-stats
  uid: ed165966-c994-11e7-871d-080027676468
spec:
  clusterIP: 10.0.0.11
  ports:
  - name: stats
    port: 56789
    protocol: TCP
    targetPort: stats
  - name: http
    port: 56790
    protocol: TCP
    targetPort: http
  selector:
    origin: voyager
    origin-api-group: voyager.appscode.com
    origin-name: test-ingress
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

@tamalsaha tamalsaha reopened this Nov 15, 2017

@jalberto

This comment has been minimized.

Copy link
Author

commented Nov 15, 2017

this sounds great! will try it soon as possible

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Nov 16, 2017

@jalberto , I will ping you when we cut the next rc with this feature.

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Feb 12, 2018

Done.

@tamalsaha tamalsaha closed this Feb 12, 2018

@tamalsaha

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.