Add optional Service and ServiceMonitor to helm chart to expose prometheus metrics

[michaeljmarshall]

Reference: aws/eks-charts#183 (comment)

The helm chart provides an option to expose prometheus metrics on a port. Prometheus metrics are often collected via a ServiceMonitor pointed at a Service, which in turn exposes the endpoints and their ports so that the prometheus server can scrape the metrics. As such, I think it'd be helpful to add an option to the helm chart that creates a ServiceMonitor and a Service to make metrics collection more easy to implement.

Let me know if you have any questions. Thanks!

[bwagner5]

@michaeljmarshall Would the service provide access to NTH in some manner or is it just used to group the pods for the ServiceMonitor?

I'm not very familiar with Prometheus, but would be open to doing this if you could clarify the moving parts or point me to some docs. I'm particularly curious about how prometheus metric pulls are affected by NTH being a daemonset. It seems like if a service was used as the ingress point, not all metrics endpoints would be scraped, but maybe I'm missing something?

[bwagner5]

Also @michaeljmarshall, what are your thoughts on the PodMonitor implemented in this PR aws/eks-charts#211 ?

[crhuber]

Added PR to create a PodMonitor #206

[michaeljmarshall]

@michaeljmarshall Would the service provide access to NTH in some manner or is it just used to group the pods for the ServiceMonitor?

I'm not very familiar with Prometheus, but would be open to doing this if you could clarify the moving parts or point me to some docs. I'm particularly curious about how prometheus metric pulls are affected by NTH being a daemonset. It seems like if a service was used as the ingress point, not all metrics endpoints would be scraped, but maybe I'm missing something?

@bwagner5 - the Prometheus Operator uses ServiceMonitors to discover the Services, and then it uses those Services to discover the Endpoints, which are then used to scrape metrics.

I am new to PodMonitors. Based on some quick reading, I think a PodMonitor makes sense here. I'll take a look at the PR and provide feedback there.

Thanks!

[bwagner5]

eks-charts has been released: https://github.com/aws/eks-charts/releases/tag/v0.0.26