Help deploying Grafana Dashboards using Generic Rules

[tech1ndex]

Hi,

First and foremost, just wanted to say that this is a fantastic project and that I thoroughly appreciate all of the work that has gone into this so far!

I was setting up a POC using the example manifests which is working great so far.

I also wanted to get the Grafana Dashboards working however I am relatively new to Grafana and feel like I am missing something obvious. I followed your instructions and created a ConfigMap containing the Generic Rules definition. I the tried to mount that ConfigMap as a volume inside the pyrra kuberentes Deployment and pass it the --generic-rules arg along with the path to the file however, the pyrra image doesn't seem to like that flag and is Crash Looping with the following message:

pyrra: error: unknown flag --generic-rules

Am I going about this the correct way?

[Rohlik]

Maybe it would be great to specify which image/chart you are using 🙄.

[tech1ndex]

Oh sorry about that, I am just using the ghcr.io/pyrra-dev/pyrra:v0.7.0 image that is deployed as part of your kubernetes examples

[Rohlik]

Are you using it correctly 🤔? I got same error only if I use it wrongly, like:

$ docker run -ti ghcr.io/pyrra-dev/pyrra:v0.7.0  --generic-rules
pyrra: error: unknown flag --generic-rules

If I use it in correct way, then there is no such an error:

$ docker run -ti ghcr.io/pyrra-dev/pyrra:v0.7.0 generate --generic-rules
level=info ts=2023-10-25T13:27:24.404827723Z caller=main.go:142 msg="using Prometheus" url=http://localhost:9090

[tech1ndex]

Yup, looks like that was it. I have an issue with the structure of my generic-rules.yaml file now it looks like :

level=info ts=2023-10-25T13:43:47.683319981Z caller=main.go:142 msg="using Prometheus" url=http://localhost:9090
level=error ts=2023-10-25T13:43:47.684517622Z caller=generate.go:33 msg="generating rule files" err="failed to get objective: failed to unmarshal objective \"/etc/pyrra/generic-rules.yaml\": error unmarshaling JSON: while decoding JSON: json: unknown field \"recordingRules\

but at least was able to progress, thanks!

[Rohlik]

The structure should look like:

apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: loki-availability
  namespace: monitoring
  labels:
    release: k8s
    role: alert-rules
spec:
  description: Loki availability based on duration of requests
  target: "99"
  window: 7d
  indicator:
    ratio:
      errors:
        metric: loki_request_duration_seconds_count{namespace="monitoring", job="loki-querier", status_code=~"5[0-9]{2}"}
      total:
        metric: loki_request_duration_seconds_count{namespace="monitoring", job="loki-querier"}

If you are happy with the solution of this issue, then you can proceed with closing this ticket.

[tech1ndex]

Awesome, thanks! Will close now.

[metalmatze]

Glad you found a solution!
--generic-rules actually only works on the filesystem, kubernetes or generate sub commands. It doesn't work on the default api command as that one only serves APIs and UIs and never generates anything.