Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically create ServiceMonitor for built-in exporter #154

Closed
tamalsaha opened this issue Jun 7, 2017 · 7 comments

Comments

@tamalsaha
Copy link
Member

commented Jun 7, 2017

To export HAProxy stats to Prometheus user can apply the following annotations to an ingress object.

monitoring.appscode.com/agent: Prometheus

monitoring.appscode.com/prometheus.Namespace: <Kube NS where service monitors will be created>
monitoring.appscode.com/prometheus.Labels: <map[string]string used to select Prometheus instance>
monitoring.appscode.com/prometheus.Interval: <scrape interval>

Voyager operator will automatically create ServiceMonitor TPR objects based on these annotations. That means if you also use Prometheus operator, your HAProxy stats will start showing up in Prometheus and Grafana Dashboard without further manual action.

@julianvmodesto

This comment has been minimized.

Copy link
Contributor

commented Jun 14, 2017

What's the best way to set up a ServiceMonitor for an ingress.appscode.com resource now that we have #13 in 1.5.6?

@tamalsaha

This comment has been minimized.

Copy link
Member Author

commented Jun 14, 2017

@julianvmodesto, the short answer is that you have to write your Prometheus Job config by hand if you want to use the /metrics endpoints exposed as part of operator. CoreOs's prometheus operator expects the exporter to be running in the same pod as the HAProxy. Otherwise, the config generated by prom operator applies incorrect labels. So, you can't use ServiceMonitor to scrape metrics for HAProxy from the operator endpoints.

Originally, I was hoping that prom operator could be extended to support out-of-pod exporters. But it seems that prom operator project does not like that idea on philosophical grounds (coreos/prometheus-operator#387) . If you are writing Prometheus config by hand, you can easily support these types of out-of-pod exporter scenarios, as I pointed out in that ticket.

Now, we are brainstorming what should be the road ahead on this, including writing our own prom operator. I would to like to know what you or any other Voyager users think on this topic?

@julianvmodesto

This comment has been minimized.

Copy link
Contributor

commented Jun 15, 2017

Ahh got it!

What's the primary benefit of an out-of-pod haproxy exporter that we'd like to preserve?

In the meantime, I'll give it a shot handwriting the scrape config with a custom ServiceMonitor.

@tamalsaha

This comment has been minimized.

Copy link
Member Author

commented Jun 15, 2017

The benefits of out-of-pod exporters as I see today are:

  • Exporter can be added / updated without restarting production workload.
  • The same exporter can be used to expose metrics for workloads of same kind (multiple postgres DB in same/diff namespace, etc.). This is particularly useful for smaller cluster.
@julianvmodesto

This comment has been minimized.

Copy link
Contributor

commented Jun 15, 2017

For our use case, we're completely okay with an in-pod exporter and losing these benefits, although it makes sense why these benefits are important for different use cases.

We're also using CoreOS's Prometheus Operator, so it would be great if we could generate a compatible ServiceMonitor resource, whether we go w/ the in-pod haproxy exporter or the out-of-pod haproxy exporter. Haven't gotten to setting up an empty serviceMonitorSelector and hand-written Prometheus config in a ConfigMap yet -- seems a bit tricky.

@tamalsaha

This comment has been minimized.

Copy link
Member Author

commented Jun 15, 2017

Supporting side-car exporters opens up a whole new can of worms regarding update s.

Also, I wonder should we have such tight support for one specific monitoring system or one operator of such system. What if some other monitoring system comes out tomorrow and that needs to be supported.

For us the issue of of out-of-pod exporter is particularly important with our DB statefulsets. I think it is unreasonable to say that you need to restart you DB so that exporters can be added/updated, where there is no technical reason for that.

@tamalsaha tamalsaha added this to the 3.0.0 milestone Jun 16, 2017

@tamalsaha

This comment has been minimized.

Copy link
Member Author

commented Jun 16, 2017

@julianvmodesto , after some internal discuss we have decided to implement this feature in next release using side-card exporters.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.