The gossip function is not working in webhook_configs.

[SEIJIIWATA]

What did you do?
As title.

What did you expect to see?
Elimination of alert duplication in prometheus and alertmanager EC2 of 2 units each configurations .

What did you see instead? Under which circumstances?
Two duplicate ALERTS were notified for one ALERT generated by prometheus.

Environment

• prometheus01
• prometheus02
• alertmanager01
• alertmanager02

• Alertmanager version:

# alertmanager --version
alertmanager, version 0.26.0 (branch: HEAD, revision: d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)
  build user:       root@df8d7debeef4
  build date:       20230824-11:11:58
  go version:       go1.20.7
  platform:         linux/amd64
  tags:             netgo

• Prometheus version:

prometheus, version 2.35.0 (branch: HEAD, revision: 6656cd29fe6ac92bab91ecec0fe162ef0f187654)
  build user:       root@cf6852b14d68
  build date:       20220421-09:53:42
  go version:       go1.18.1
  platform:         linux/amd64

• Alertmanager configuration file:

global:
  resolve_timeout: 5m
  smtp_from: '###SMTP_FROM###'
  smtp_smarthost: '###SMTP_HOST###:10587'
  smtp_auth_username: '###SMTP_USER###'
  smtp_auth_password: '###SMTP_PASSWORD###'

templates:
  - templates/*.tmpl

# don't use this rules basically
inhibit_rules:

route:
  receiver: test0
  repeat_interval: 3h
  routes:
  - receiver: test0
    continue: true
  - match:
      system_id: test1
      subsystem_id: test1
    receiver: test1

receivers:
- name: 'test0'
  slack_configs:
  - channel: '#******'
    api_url: '*****'
    title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
    text: "{{ range .Alerts }} \nAdditional Info: {{ .Annotations.description }} \nAddress:{{ .Labels.instance }} \nName:{{ .Labels.name }} \nState:{{ .Labels.severity }}\n{{ end }}"
    send_resolved: true
- name: 'test1'
  webhook_configs:
  - url: 'http://localhost:19000/hooks/TEST'
    send_resolved: true

• Prometheus configuration file:

global:
  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 1m
  external_labels:
    region: ap-northeast-1
    replica: ###AZ###

alerting:
  alert_relabel_configs:
  - regex: replica
    action: labeldrop
  alertmanagers:
  - timeout: 10s
    api_version: v2
    static_configs:
    - targets:
      - alertmanager01.test:9093
      - alertmanager02.test:9093

rule_files:
- rules/rule_*.yml

• Stat setting of alert manager:
  ※alertmanager01

/usr/bin/alertmanager --config.file=/usr/local/alertmanager/config/alertmanager.yml --web.external-url=http://alertmanager01.test --web.route-prefix=/ --cluster.peer=alertmanager02.test:9094

● alertmanager.service - alertmanager service
   Loaded: loaded (/usr/lib/systemd/system/alertmanager.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2024-08-26 07:33:09 UTC; 6s ago
 Main PID: 1453 (alertmanager)
   CGroup: /system.slice/alertmanager.service
           └─1453 /usr/bin/alertmanager --config.file=/usr/local/alertmanager/config/alertmanager.yml --web.external-url=http://alertmanager01.test --web.route-prefix=/ --cluster.peer=alertmanager02.test:9094

※alertmanager02

/usr/bin/alertmanager --config.file=/usr/local/alertmanager/config/alertmanager.yml --web.external-url=http://alertmanager02.test --web.route-prefix=/ --cluster.peer=alertmanager01.test:9094

● alertmanager.service - alertmanager service
   Loaded: loaded (/usr/lib/systemd/system/alertmanager.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2024-08-26 07:33:09 UTC; 6s ago
 Main PID: 1453 (alertmanager)
   CGroup: /system.slice/alertmanager.service
           └─1453 /usr/bin/alertmanager --config.file=/usr/local/alertmanager/config/alertmanager.yml --web.external-url=http://alertmanager02.test --web.route-prefix=/ --cluster.peer=alertmanager01.test:9094