Prometheus rule created but not loaded on prometheus operator

[stafot]

Hello,
I am not sure if this is a bug or I am missing something in setup process.

I am trying to setup sloth on my cluster based on this approach https://github.com/slok/sloth#kubernetes-controller-prometheus-operator.
I successfully set up the sloth controller which runs without any errors.
I added a sample slo for a service running. The respective prometheus rule created successfully. But the rule doesn't get attached to prometheus operator. Should I do it manually? If yes why not to follow the cli approach instead?

Thanks in advance for any answer.

[slok]

Hi @stafot

Once the Prometheus operator Rules CRs are created, Sloth's job finishes, and you shouldn't need to do anything manually either.

Are you already loading automatically any other rules (apart from the Sloth ones) with the Prometheus operator?

Normally when you create a Prometheus instance with the Prom-operator, you will tell what is the label selector this Prometheus needs to match to discover the rules it should own (and load), if you don't have this configured correctly it will not discover the created rules. For example:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
  labels:
    prometheus: prometheus
spec:
# ...
  ruleSelector:
    matchLabels:
      prometheus: prometheus
      role: my-awesome-rules
# ...

Now my PrometheusRules CRs at least should have prometheus: prometheus and role: my-awesome-rules labels so this prom can load them automatically. Sloth will set the same labels the Sloth CRs has to the PrometheusRule CR, so, for example:

---
apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: some-slo
  namespace: monitoring
  labels:
    prometheus: prometheus
    role: my-awesome-rules
    app: sloth
    something: something
spec:
# ...

Let me know if this solves the problem.

[stafot]

Thanks for your quick answer. I will review my setup based on your instructions and I will come back to you! Appreciated.

[stafot]

@slok Thank you very much, your suggestion did the trick.

[slok]

Awesome!