How to disable Watchdog initial notification in Alertmanager

[ahmedTouati]

I installed Alertmanager in my EKS cluster along with Prometheus and set up some alerts, they all working fine except one annoying alert that spin up every time which is the Watchdog notification that tells that the entire pipeline is working fine, I know it's an important alert but we have one receiver that accepts all kind of alerts, and it's really annoying to get notified at 12pm to only see that one alert i tried to get rid of it by redirecting it to a null receive but it doesn't work.

What did you expect to see?
Disable the Watchdog alert

What did you see instead?
The Watchdog alert keeps firing all the time

Environment

• System information:

  EKS cluster v1.16

• Alertmanager version:

  v0.21.0

• Prometheus version:

  v2.21.0

• The Watchdog alert*

alert: Watchdog
expr: vector(1)
labels:
  severity: none
annotations:
  message: |
    This is an alert meant to ensure that the entire alerting pipeline is functional.
    This alert is always firing, therefore it should always be firing in Alertmanager
    and always fire against a receiver. There are integrations with various notification
    mechanisms that send a notification when this alert is not firing. For example the
    "DeadMansSnitch" integration in PagerDuty.

• Alertmanager configuration file:

config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['job']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: prometheus-msteams
      routes:
      - match:
          alertname: Watchdog
        receiver: prometheus-msteams
    receivers:
    - name: prometheus-msteams
      webhook_configs:
      - url: "http://prometheus-msteams:2000/alertmanager"
        send_resolved: true

[zhmurko]

try to add an additional empty receiver and route Watchdog alert to it. Mind the order of the rules.

config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['job']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      routes:
      - receiver: "empty"
        match:
          alertname: Watchdog
      - receiver: "prometheus-msteams"
    receivers:
    - name: "empty"
    - name: prometheus-msteams
      webhook_configs:
      - url: "http://prometheus-msteams:2000/alertmanager"
        send_resolved: true

[simonpasquier]

You can route the Watchdog alert to an empty receiver as @zhmurko suggested.

I'm closing it for now. If you have further questions, please use our user mailing list, which you can also search.

[tusharsonawanes]

Any idea how to disable this now?

[yavorivanov-cw]

We managed to do it with (after many many fails)

  route:
    groupBy: ['namespace']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: 'slack-notification'
    routes:
      - receiver: 'null'
        matchers:
        - matchType: "="
          name: alertname
          value: Watchdog
  receivers:
  - name: 'null'
  - name: 'slack-notification'
    slackConfigs:
    ...

We tried to use matchType: "!=" without a null receiver but it didnt work.