got duplicate alerts with the alertmanager 0.12.0 #3755

likunbyl · Jan 28, 2018

What did you do?
I'm deploying prometheus at k8s environment. two prometheus pods, point to two alertmanager pods, with mesh.peer point to each other.

Then I stopped a monitored service, got duplicate alert.

What did you expect to see?
deduplicate alert successfully

What did you see instead? Under which circumstances?
I stopped a monitered service with the command:
# kubectl scale deploy spark-master-1 --replicas=0 -nkube-system

got two warning fires:

Environment

System information:
Linux 4.9.9-coreos-r1 x86_64
Prometheus version:
prometheus, version 1.7.1 (branch: master, revision: 3afb3ff)
build user: root@0aa1b7fc430d
build date: 20170612-11:44:05
go version: go1.8.3
Alertmanager version:
alertmanager, version 0.12.0 (branch: HEAD, revision: fc33cc78036f82ef8d4734c197a96f7cb6c952a3)
build user: root@c9169eb10d06
build date: 20171215-14:13:20
go version: go1.9.2
Prometheus configuration file:

The master prometheus-deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-m
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-m
    spec:
      containers:
      - name: prometheus-m
        image: prometheus/prometheus:v1.7.1
        args:
        - -storage.local.memory-chunks=1000000000
        - -storage.local.index-cache-size.fingerprint-to-metric=30485760
        - -storage.local.index-cache-size.label-name-to-label-values=20485760
        - -storage.local.index-cache-size.label-pair-to-fingerprints=30971520
        - -storage.local.retention=8760h
        - -storage.local.path=/prometheus
        - -storage.local.chunk-encoding-version=2
        - -web.console.libraries=/usr/share/prometheus/console_libraries
        - -web.console.templates=/usr/share/prometheus/consoles
        - -config.file=/etc/prometheus/prometheus.yml
        - -web.external-url=http://prometheus-m.wjs.lab.com
        ports:
        - containerPort: 9090
          hostPort: 9090
          name: web
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/localtime
          name: timezone
        - mountPath: /prometheus
          name: prometheus-storage
        - mountPath: /etc/prometheus
          name: config-volume
      nodeSelector:
        kube-system-prometheus: node1
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      serviceAccount: prometheus
      volumes:
      - hostPath:
          path: /etc/localtime
        name: timezone
      - hostPath:
          path: /data1/prometheus
        name: prometheus-storage
      - configMap:
          name: prometheus-m
        name: config-volume

The slave prometheus-deployment.yaml

kind: Deployment
metadata:
  name: prometheus-s
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-s
    spec:
      containers:
      - name: prometheus-s
        image: prometheus/prometheus:v1.7.1
        args:
        - -storage.local.memory-chunks=1000000000
        - -storage.local.index-cache-size.fingerprint-to-metric=30485760
        - -storage.local.index-cache-size.label-name-to-label-values=20485760
        - -storage.local.index-cache-size.label-pair-to-fingerprints=30971520
        - -storage.local.retention=8760h
        - -storage.local.path=/prometheus
        - -storage.local.chunk-encoding-version=2
        - -web.console.libraries=/usr/share/prometheus/console_libraries
        - -web.console.templates=/usr/share/prometheus/consoles
        - -config.file=/etc/prometheus/prometheus.yml
        - -web.external-url=http://prometheus-s.wjs.lab.com
        ports:
        - containerPort: 9090
          hostPort: 9090
          name: web
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/localtime
          name: timezone
        - mountPath: /prometheus
          name: prometheus-storage
        - mountPath: /etc/prometheus
          name: config-volume
      hostNetwork: true
      nodeSelector:
        kube-system-prometheus: node2
      dnsPolicy: ClusterFirstWithHostNet
      serviceAccount: prometheus
      volumes:
      - hostPath:
          path: /etc/localtime
        name: timezone
      - hostPath:
          path: /data1/kube-system/prometheus
        name: prometheus-storage
      - configMap:
          name: prometheus-s
        name: config-volume


The prometheus.yml:

global:
  scrape_interval: 180s
  scrape_timeout: 60s
  evaluation_interval: 60s
rule_files:
  - up.rules

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - alertmanager-m:9093
      - alertmanager-s:9093

scrape_configs:


- job_name: 'kubernetes-services'
  scrape_interval: 20s
  metrics_path: /probe
  params:
    module: [http_2xx]
  kubernetes_sd_configs:
  - role: service
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
    action: keep
    regex: true
  - source_labels: [__address__]
    target_label: __param_target
  - source_labels: [__param_target, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    regex: (.+):(?:\d+);(.+)
    replacement: $1:$2
    target_label: __param_target
  - source_labels: [__param_target, __meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    regex: (.+);(.+)
    replacement: ${1}/${2}
    target_label: __param_target
  - target_label: __address__
    replacement: 192.168.194.122:9115
  - source_labels: [__param_target]
    target_label: instance
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    target_label: kubernetes_name


The up.rules:


    ALERT NodeDown
      IF up {job="kubernetes-nodes" } == 0
      FOR 10m
      LABELS { severity = "wjs-lab" , sender = "email,sms" , service="PEK"}
      ANNOTATIONS {
        summary = "node {{ $labels.instance }} is down ",
        description = "{{ $labels.instance }}  has been down for more than 10 minutes",
      }
    ALERT PodDown
      IF up {job="kubernetes-service-endpoints", kubernetes_namespace !~ ".*(dev|pre|gps|ulab)$" } == 0
      FOR 10m
      LABELS { severity = "wjs-lab"  , sender = "email,sms" , service="PEK"  }
      ANNOTATIONS {
        summary = "{{ $labels.kubernetes_name }} is down ",
        description = "{{ $labels.kubernetes_name }} of {{ $labels.kubernetes_namespace }}  has been down for more than 10 minutes",
      }
    ALERT ApiServerDown
      IF up {job="kubernetes-apiservers" } == 0
      FOR 10m
      LABELS { severity = "wjs-lab"  , sender = "email,sms" , service="PEK" }
      ANNOTATIONS {
        summary = "{{ $labels.instance }} is down ",
        description = "{{ $labels.instance }}  has been down for more than 10 minutes",
      }
    ALERT ServiceDown
      IF probe_success {job="kubernetes-services", kubernetes_namespace !~ ".*(dev|pre|gps|ulab)$" } == 0
      FOR 10m
      LABELS { severity = "wjs-lab"  , sender = "email,sms" , service="PEK"  }
      ANNOTATIONS {
        summary = "{{ $labels.kubernetes_name }} is down ",
        description = "{{ $labels.kubernetes_name }} of {{ $labels.kubernetes_namespace }}  has been down",
      }

Alertmanager configuration file:

The master alertmanager-deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: alertmanager-m
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: alertmanager-m
    spec:
      containers:
      - args:
        - -log.level=debug
        - -mesh.peer=alertmanager-s.kube-system.svc:6783
        - -config.file=/etc/alertmanager/alertmanager.yml
        image: prometheus/alertmanager:v0.12.0
        name: alertmanager
        ports:
        - containerPort: 9093
          hostPort: 9093
          name: alertmanager
          protocol: TCP
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "1"
            memory: 1Gi
        volumeMounts:
        - mountPath: /etc/alertmanager
          name: config-volume
        - name: timezone
          mountPath: /etc/localtime
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      volumes:
      - configMap:
          name: alertmanager
        name: config-volume
      - hostPath:
          path: /etc/localtime
        name: timezone

The slave alertmanager-deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: alertmanager-s
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: alertmanager-s
    spec:
      containers:
      - args:
        - -log.level=debug
        - -config.file=/etc/alertmanager/alertmanager.yml
        - -mesh.peer=alertmanager-m.kube-system.svc:6783
        image: prometheus/alertmanager:v0.12.0
        name: alertmanager
        ports:
        - containerPort: 9093
          hostPort: 9093
          name: alertmanager
          protocol: TCP
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "1"
            memory: 1Gi
        volumeMounts:
        - mountPath: /etc/alertmanager
          name: config-volume
        - name: timezone
          mountPath: /etc/localtime
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      volumes:
      - configMap:
          name: alertmanager
        name: config-volume
      - hostPath:
          path: /etc/localtime
        name: timezone

The alertmanger configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager
  namespace: kube-system
data:
  alertmanager.yml: |-
    global:
    route:
     group_by: ['alertname', 'app', 'kubernetes_name', 'kubernetes_namespace']
     group_wait: 1m
     group_interval: 5m
     repeat_interval: 35m
     receiver: webhook-team-base

     routes:
      - match:
          severity: wjs-lab
        receiver: webhook-team-base

      - match:
          severity: pek-base
        receiver: webhook-team-base

      - match:
          severity: pek-important
        receiver: webhook-team-important

    receivers:
    - name: webhook-team-base
      webhook_configs:
      - url: http://pekapi.corp.com/alarm

    - name: webhook-team-important
      webhook_configs:
      - url: http://pekapi.corp.com/alarm

Logs:
my local time is 8 hours behind the utc time.

The logs of alertmanager-m:

level=info ts=2018-01-14T03:16:15.455027317Z caller=main.go:155 msg="Starting Alertmanager" version="(version=0.12.0, branch=HEAD, revision=fc33cc78036f82ef8d4734c197a96f7cb6c952a3)"
level=info ts=2018-01-14T03:16:15.45510958Z caller=main.go:156 build_context="(go=go1.9.2, user=root@c9169eb10d06, date=20171215-14:13:20)"
level=info ts=2018-01-14T03:16:16.304228938Z caller=main.go:293 msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml
level=info ts=2018-01-14T03:16:16.335245199Z caller=main.go:368 msg=Listening address=:9093
ts=2018-01-14T03:16:21.52604898Z caller=<autogenerated>:1 component=mesh level=debug msg="->[10.3.7.33:52357] connection accepted"
ts=2018-01-14T03:16:21.526864234Z caller=<autogenerated>:1 component=mesh level=debug msg="->[10.3.7.33:52357|14:18:77:58:b1:16(az05.wjs.lab.com)]: connection ready; using protocol version 2"
ts=2018-01-14T03:16:21.526929054Z caller=<autogenerated>:1 component=mesh level=debug msg="->[10.3.7.33:52357|14:18:77:58:b1:16(az05.wjs.lab.com)]: connection added (new peer)"
level=debug ts=2018-01-14T03:28:37.807027934Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:29:37.807563901Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"wjs-lab\"}:{alertname=\"ServiceDown\", app=\"spark-master-1\", kubernetes_name=\"spark-webui-1\", kubernetes_namespace=\"kube-system\"}" msg=Flushing alerts=[ServiceDown[c3768e6][active]]
level=debug ts=2018-01-14T03:29:37.808493962Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:29:37.808547381Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:29:50.934530065Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"wjs-lab\"}:{alertname=\"ServiceDown\", app=\"spark-master-1\", kubernetes_name=\"spark-webui-1\", kubernetes_namespace=\"kube-system\"}" msg=Flushing alerts=[ServiceDown[c3768e6][active]]
level=debug ts=2018-01-14T03:30:37.808608791Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:30:37.809078745Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=info ts=2018-01-14T03:31:15.459343982Z caller=nflog.go:287 component=nflog msg="Running maintenance"
level=info ts=2018-01-14T03:31:15.459496486Z caller=silence.go:262 component=silences msg="Running maintenance"
level=info ts=2018-01-14T03:31:15.468448872Z caller=silence.go:279 component=silences msg="Maintenance done" duration=50.377Âµs
level=info ts=2018-01-14T03:31:15.500600509Z caller=nflog.go:304 component=nflog msg="Maintenance done" duration=92.53Âµs
level=debug ts=2018-01-14T03:31:37.808071079Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:31:37.808149879Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:32:37.808537737Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:32:37.808622443Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:33:37.832905526Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:33:37.832985134Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:34:37.807017396Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:34:37.807919117Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:34:50.934800604Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"wjs-lab\"}:{alertname=\"ServiceDown\", app=\"spark-master-1\", kubernetes_name=\"spark-webui-1\", kubernetes_namespace=\"kube-system\"}" msg=Flushing alerts=[ServiceDown[c3768e6][active]]
level=debug ts=2018-01-14T03:35:37.808244677Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:35:37.808328959Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:36:37.807302257Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:36:37.808440073Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:37:37.807359156Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:37:37.808236539Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:38:37.807127924Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:38:37.808280207Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:39:37.806889548Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:39:37.806959233Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:39:50.935460608Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"wjs-lab\"}:{alertname=\"ServiceDown\", app=\"spark-master-1\", kubernetes_name=\"spark-webui-1\", kubernetes_namespace=\"kube-system\"}" msg=Flushing alerts=[ServiceDown[c3768e6][active]]
level=debug ts=2018-01-14T03:40:37.807365948Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:40:37.808391223Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]


The logs of alertmanager-s:

level=info ts=2018-01-14T03:16:21.521054945Z caller=main.go:155 msg="Starting Alertmanager" version="(version=0.12.0, branch=HEAD, revision=fc33cc78036f82ef8d4734c197a96f7cb6c952a3)"
level=info ts=2018-01-14T03:16:21.521117681Z caller=main.go:156 build_context="(go=go1.9.2, user=root@c9169eb10d06, date=20171215-14:13:20)"
level=info ts=2018-01-14T03:16:21.525369093Z caller=main.go:293 msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml
ts=2018-01-14T03:16:21.525427641Z caller=<autogenerated>:1 component=mesh level=debug msg="->[10.3.7.34:6783] attempting connection"
ts=2018-01-14T03:16:21.526741844Z caller=<autogenerated>:1 component=mesh level=debug msg="->[10.3.7.34:6783|14:18:77:59:82:b7(az06.wjs.lab.com)]: connection ready; using protocol version 2"
ts=2018-01-14T03:16:21.526808118Z caller=<autogenerated>:1 component=mesh level=debug msg="->[10.3.7.34:6783|14:18:77:59:82:b7(az06.wjs.lab.com)]: connection added (new peer)"
level=info ts=2018-01-14T03:16:21.531705353Z caller=main.go:368 msg=Listening address=:9093
level=debug ts=2018-01-14T03:28:37.808185416Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:29:37.808387503Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:29:37.808478452Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:29:37.808544044Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"wjs-lab\"}:{alertname=\"ServiceDown\", app=\"spark-master-1\", kubernetes_name=\"spark-webui-1\", kubernetes_namespace=\"kube-system\"}" msg=Flushing alerts=[ServiceDown[c3768e6][active]]
level=debug ts=2018-01-14T03:29:47.929030466Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"wjs-lab\"}:{alertname=\"ServiceDown\", app=\"spark-master-1\", kubernetes_name=\"spark-webui-1\", kubernetes_namespace=\"kube-system\"}" msg=Flushing alerts=[ServiceDown[c3768e6][active]]
level=debug ts=2018-01-14T03:30:37.808150843Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:30:37.808375736Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=info ts=2018-01-14T03:31:21.521776766Z caller=silence.go:262 component=silences msg="Running maintenance"
level=info ts=2018-01-14T03:31:21.521928415Z caller=nflog.go:287 component=nflog msg="Running maintenance"
level=info ts=2018-01-14T03:31:21.560475286Z caller=silence.go:279 component=silences msg="Maintenance done" duration=477.532Âµs
level=info ts=2018-01-14T03:31:21.585094362Z caller=nflog.go:304 component=nflog msg="Maintenance done" duration=452.972Âµs
level=debug ts=2018-01-14T03:31:37.806716907Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:31:37.806837411Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:32:37.808293015Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:32:37.808402775Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:33:37.806951218Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:33:37.80795076Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:34:37.806845029Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:34:37.807656797Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:34:47.929238209Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"wjs-lab\"}:{alertname=\"ServiceDown\", app=\"spark-master-1\", kubernetes_name=\"spark-webui-1\", kubernetes_namespace=\"kube-system\"}" msg=Flushing alerts=[ServiceDown[c3768e6][active]]
level=debug ts=2018-01-14T03:35:37.806542696Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:35:37.806633492Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:36:37.808177879Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:36:37.808256432Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:37:37.860991979Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:37:37.861074503Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:38:37.806660256Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:38:37.808161099Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:39:37.80664663Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:39:37.808074514Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:39:47.929490379Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"wjs-lab\"}:{alertname=\"ServiceDown\", app=\"spark-master-1\", kubernetes_name=\"spark-webui-1\", kubernetes_namespace=\"kube-system\"}" msg=Flushing alerts=[ServiceDown[c3768e6][active]]
level=debug ts=2018-01-14T03:40:37.806969853Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]
level=debug ts=2018-01-14T03:40:37.808265096Z caller=dispatch.go:188 component=dispatcher msg="Received alert" alert=ServiceDown[c3768e6][active]

krasi-georgiev · Jan 28, 2018

shouldn't this be posted in https://github.com/prometheus/alertmanager ?

likunbyl · Jan 29, 2018

ok, I'll repost it there.

lock · Mar 22, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

likunbyl closed this Jan 29, 2018

lock bot locked and limited conversation to collaborators Mar 22, 2019

prometheus/prometheus

got duplicate alerts with the alertmanager 0.12.0 #3755

got duplicate alerts with the alertmanager 0.12.0 #3755

likunbyl commented Jan 28, 2018 •

edited

This comment has been minimized.

krasi-georgiev commented Jan 28, 2018

This comment has been minimized.

likunbyl commented Jan 29, 2018

likunbyl closed this Jan 29, 2018

This comment has been minimized.

lock bot commented Mar 22, 2019

lock bot locked and limited conversation to collaborators Mar 22, 2019

prometheus/prometheus

Join GitHub today

got duplicate alerts with the alertmanager 0.12.0 #3755

Comments

likunbyl commented Jan 28, 2018 • edited

This comment has been minimized.

krasi-georgiev commented Jan 28, 2018

This comment has been minimized.

likunbyl commented Jan 29, 2018

likunbyl closed this Jan 29, 2018

This comment has been minimized.

lock bot commented Mar 22, 2019

lock bot locked and limited conversation to collaborators Mar 22, 2019

likunbyl commented Jan 28, 2018 •

edited