Monitoring Custom Metrics with Grafana

[osterman]

what

• Ability to view metrics/graphs/reports
• How do we add custom metrics to graphs?

See #20 (requested by @Nuru)

[osterman]

@Nuru can you help me scope this question?

Are you asking about custom application metrics or metrics already available, but not exposed through any dashboard.

If we're talking about custom application metrics (e.g. open comments), that's a much larger discussion as we need to go into how to expose metrics for each app and then harvest them with Prometheus.

It would help if you give some example metrics off the top of your head so that we can address it appropriately.

[Nuru]

@osterman I'm not asking how to generate metrics inside an app, I'm expecting that is generically documented somewhere. I'm asking how do we connect custom metrics we are generating to Prometheus and then Grafana.

For example, let's say we wanted to track the number of new user signups per day. Please point me to generic documentation that describes Prometheus' requirements for an endpoint that provides that data and then how (with URLs and/or file names) to connect that endpoint to Prometheus and then add that widget to the Grafana dashboard.

I'm only asking for the PopChest specific data that needs to be merged with generic documentation to accomplish the task. The answer could be as simple as

• Add the REST endpoint for a Prometheus data source meeting the requirements described at <public generic Prometheus documentation URL> to the Prometheus agent via the <menu item name> at the <control panel URL for staging> and/or <control panel URL for production>. Give it a name using <this field>.
• Create a Grafana widget via <menu item> at <URL> and reference the name you gave the data source in the previous step.

[osterman]

This has been partially addressed here: https://docs.cloudposse.com/kubernetes-backing-services/monitoring/prometheus-alerts-grafana/