Install Prometheus and Grafana with eoAPI

[ranchodeluxe]

Background

To do HPA/VPA properly we need metrics and Prometheus seems to be the best game in town. We also need to visualize metrics to understand whats happening with resourcing in the clstuer. Grafana seems to be the most used open-source project for this.

While these two packages can easily be helm installed the main purpose of this ticket is to help users get started quick with docs and other things below

AC:

• write some documentation about how users of eoapi-k8s can install Prometheus and Grafana
• is there someway we can export a set of basic Grafana starter graphs?
• we'll want this as a separate chart just like 2i2c does for these "dependency" patterns

[Rub21]

[ ] write some documentation about how users of eoapi-k8s can install Prometheus and Grafana

Updates

Prometheus:

Prometheus is a monitoring tool that helps you keep track of various metrics and statistics for your computer systems, applications, and services. It collects data over time(stores 15days), allowing you to analyze performance, troubleshoot issues.

Exporters: https://prometheus.io/docs/instrumenting/exporters/

Exporters are programs that collect metrics from a specified source and transform them into a format that can be consumed by Prometheus.

• Node_exporter

The node_exporter is designed to collect metrics from the host system, for Unix-like systems.
By default, the installation of Prometheus includes node_exporter. This exporter gathers system information such as CPU usage, memory, disk usage, and more, and presents it in a format that Prometheus can collect.

• cAdvisor

cAdvisor (Container Advisor) is an open-source monitoring tool for containers. it provides detailed information about resource usage and performance characteristics of running containers.

Installing Prometheus, Grafana and exporters:

Clone the repo: https://github.com/Rub21/k8s-monitoring

To access the Grafana dashboard, you'll need to set up an environment variable. For example, use the command export GRAFANA_ADMIN_PASSWORD=abcd. By default, the password is set to 1234.

./deploy.sh create
# tab1
kubectl port-forward svc/prometheus-operated 9090 --namespace prometheus
# tab2
kubectl port-forward deployment/prometheus-grafana 3000 --namespace prometheus

Once the UI dashboards are exported, you can open them.

Prometheus dashboard

http://localhost:9090/targets?search=

Grafana dashboard

• Node exporter
  http://localhost:3000/d/rYdddlPWk/node-exporter-full?orgId=1

• cAdvisor

The dashboards can be customized for each specific use case. There are still more metrics to explore that could be useful or necessary for the EOAPI scenario.

[Rub21]

is there someway we can export a set of basic Grafana starter graphs?

@ranchodeluxe , Regarding your statement, are you suggesting that we should export the charts as PNG images? there is this grafana-image-renderer that we may need to install in the cluster to export the images,

Alternatively, if you're referring to exporting the chart data, it can be downloaded as a CSV file.

[ranchodeluxe]

is there someway we can export a set of basic Grafana starter graphs?

@ranchodeluxe , Regarding your statement, are you suggesting that we should export the charts as PNG images? there is this grafana-image-renderer that we may need to install in the cluster to export the images

Nice work @Rub21 🥳

Grafana allows us to export dashboards as JSON so they can be imported by downstream users. I've been building a basic set of eoAPI dashboards that actually helped me last week diagnose that our stac pods were being throttled and killed b/c they were running into CPU limits in the integration test pipeline. So I can handle this second AC item. Let me leave a separate comment below on your AC 1&3 stuff.

[ranchodeluxe]

[ ] write some documentation about how users of eoapi-k8s can install Prometheus and Grafana

Updates

Prometheus:

Prometheus is a monitoring tool that helps you keep track of various metrics and statistics for your computer systems, applications, and services. It collects data over time(stores 15days), allowing you to analyze performance, troubleshoot issues.

Exporters: https://prometheus.io/docs/instrumenting/exporters/

This is a great start @Rub21 💯 A couple comments.

• For AC 1 can you move your documentation into a Markdown file into the /docs folder please and add a link to the README TOC (in the future we'll probably want to render a whole GH pages site for docs but this is fine for now)

• Not sure we really need a separate repo with a bash script that calls helm install to add these things. If you look at the link in AC 3 it shows how JH does similar things in the same repo. helm can have deps and that pattern will make more sense for the community I beleive. Are you comfortable mimicing that? Or we can jam on it together if needed? Let me know