-
|
I'm currently using kube-prometheus to build and modify the prometheus alerts and rules. I now also need to add some additional custom alerts of my own however there seems to be a limitation of doing this in jsonnet in the same file. Here is my current libsonnet file: Here is my build.sh (modified a few things): For my custom rules, i created it as a yaml file and then i convert it to json using the following command:
File contents: prometheus-rules/custom-prometheus-rules.yaml When I run the following command, I'm presented with the following error:
Error: I'm aware that this is happening because in my ukiss-alerts.libsonnet I am specifying prometheusAlerts twice, however I can't seem to find a way to add my new rules but also modify the default rules at the same time. Ideally, I'd want to add my new rules in first via: And once my new rules are added, I want to loop through all of the rules and modify them via: Anyone able to help? Side note: If someone has a better suggestion of how I'm adding new labels to the default alerts, please let me know. I'm aware my if, else if statements are a bit long but it's the only way i know how. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Yeah this is a jsonnet thing, you need to have an additional merge operation to do both the local kp =
(import 'kube-prometheus/kube-prometheus.libsonnet') +
{
_config+:: {
namespace: 'monitoring',
},
prometheusAlerts+:: {
groups+: (import 'prometheus-rules/custom-prometheus-rules.json').groups,
},
grafanaDashboards+:: { // monitoring-mixin compatibility
'multi-cluster.json': (import 'grafana-dashboards/multi-cluster-dashboard.json'),
},
grafana+:: {
dashboards+:: { // use this method to import your dashboards to Grafana
'multi-cluster.json': (import 'grafana-dashboards/multi-cluster-dashboard.json'),
},
// Adds the grafana_datasource metadata label to all grafana dashboards so they can be autodiscovered by grafana
dashboardDefinitions+: {items: std.map(
function(cm) cm + {metadata+:{labels+: {grafana_datasource: 'true'}}},
super.items)},
},
} + {
prometheusAlerts+:: {
groups: std.map(
function(group)
group {
rules: std.map(
function(rule)
if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "warning" then
rule {
labels: {
application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
severity: "P4",
sparkGroup: std.extVar("SPARK_GROUP"),
affectedCi: "{{ $labels.spark_ci }}",
},
}
else if std.objectHas(rule, 'labels') && std.objectHas(rule.labels, 'severity') && rule.labels.severity == "critical" then
rule {
labels: {
application: "{{ if $labels.service }} {{ $labels.service }} {{ else }} UKISS GKE {{ end }}",
severity: "P3",
sparkGroup: std.extVar("SPARK_GROUP"),
affectedCi: "{{ $labels.spark_ci }}",
},
}
else
rule,
group.rules
),
},
super.groups
),
},
};
{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }We tend to organize patches like the one you have for changing alert labels in separate files, that way it's much easier to keep your main file organized. Let me know if this works for you! :) |
Beta Was this translation helpful? Give feedback.
-
|
The above worked great! Really appreciate your help with this. Thank you. |
Beta Was this translation helpful? Give feedback.
Yeah this is a jsonnet thing, you need to have an additional merge operation to do both the
mapand adding. Something along the lines of this should work: