Question: Handling DeadmanSwitch in Karma

[felskrone]

I monitor several services with prometheus and set up a DeadmanSwitch to ensure my alerting pipeline is working.

The DeadmanSwitch is an active alert that is always firing:

      #
      # Constant testing of the alerting pipeline
      #
      - alert: DeadMansSwitch
        expr: vector(1)
        labels:
          severity: clear
          team: MyTeam
          hive: DeadManSwitch
        annotations:
          description: vector(1)
          title: "DeadManSwitch ServerA"

Since its an active alarm it always shows up in Karma. But what im actually interested in is, to see if the alert is missing.

Is this possible with karma? I could not find anything in the documentation that occured to me as an approach to this.

I would not expect anything to be there as its a dashboard for showing alerts, but i figured asking wouldnt hurt :)

[prymitive]

I'm not sure what you expect karma to do here. Did you consider using something like https://github.com/gouthamve/deadman to generate alerts when your DeadMansSwitch is now being generated?

[felskrone]

I would like to have a DeadmanSnitch without an external service like the one you linked to.

So basically have the possibility to "invert" an alert and report if it is missing instead of showing in the dashboard.

Just like the absent() function in prometheus for testing if a metric is missing.

[prymitive]

I think it’s worth adding a health check feature like that. Will look into this, thanks.

[prymitive]

Raised #2546 that adds new config options for it

[felskrone]

Cool, thx!