Apply Prometheusrule from custom rules file and reload it upon modifications to custom alert rules file

[nitin-ag]

No description provided.

[nitin-ag]

Hello,
We are attempting prometheus and Alertmanager install using prometheus operator. Our design approach is

1. To extract all metric based alert rules in one yaml file from our application.
2. Use the yaml file above and convert those rules into prometheusrules and apply them to prometheus.
3. If custom rules file from our application sees some modifications ( add or remove of alerts) then should reload the prometheusrules for prometheus without having to restart the container.

Long story short , idea is to produce alert rules for prometheus from our application and deploy them at runtime and reload at runtime upon ant change.

Wondering what could be reasonable way to achieve above use case ?

Any suggestions would be much appreciated.

Thanks.

[pgier]

Prometheus operator should be fine for you use case as described. As you create/update PrometheusRule objects containing your alerting and recording rules, prometheus operator will pick up these changes and apply them to matching Prometheus instances.

[nitin-ag]

Thank you for your response.
But what is unclear to me is

1. Is there a way to create Prometheusrules object from our application alerts yaml File ? What should be the format of the yaml file we produce etc ?
2. Once Adjust config generation for Prometheus v1.3.0-beta.0 #1 is done and lets say our application updates the source alert file ? How to update the prometheus rules object deployed into prometheus container.

Sorry If I m asking some dumb questions here but being a new bee to these cloud technologies still in learning phase.

[pgier]

1. PrometheusRules is a kubernetes custom resource, so you can create it the same way you'd create any resource in kubernetes (i.e. kubectl apply, golang client, etc.). You can see an example of the format in the kube-prometheus project (https://github.com/coreos/kube-prometheus/blob/master/manifests/prometheus-rules.yaml).
2. The Prometheus operator reads the k8s api to find Prometheus custom resources, and then deploys Promtheus containers based on those resources. It also uses the API server to watch for changes to PrometheusRule resources and when changes are detected it will update the Prometheus config file, and tell the Prometheus instance to reload the config.

If you are new to Kubernetes, it might be easier to just create a simple Prometheus deployment at first before using prometheus-operator. Then take a look at the kube-prometheus project (https://github.com/coreos/kube-prometheus) for a more detailed example of a deployment with prometheus-operator.

[stale]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

[paulfantom]

This issue hasn't been updated for a longer period of time and appears to be answered. If you need additional clarification, please reopen it.