Alerts are not found in Prometheus during upgrade

[dhbarman]

Proposal

Prometheus triggers alarms even though there is no triggering event according to history.
This happens during Prometheus upgrade from 2.3.0 to 2.4.3.

Bug Report

What did you do?

Upgraded Prometheus version from 2.3.0 to 2.4.3 in-place.

What did you expect to see?
No alarms getting triggered.

What did you see instead? Under which circumstances?
Alarm got triggered even though there is no incidence.

Environment

• System information:

  Linux 4.1.12-124.20.7.el7uek.x86_64 x86_64

• Prometheus version:

prometheus, version 2.4.3 (branch: heads/v2.4.3, revision:
163592e656645ef8efa5458ab8af0465d6048486)
build user: teamcity@tca-hv02-agent012-default
build date: 20181110-20:51:40
go version: go1.11.2

• Alertmanager version:

  alertmanager, version 0.14.0 (branch: HEAD, revision: 30af4d051b37ce817ea7e35b56c57a0e2ec9dbb0)
  build user: root@37b6a49ebba9
  build date: 20180213-08:16:42
  go version: go1.9.2

• Prometheus configuration file:

- alert: BGP_PEER_DOWN
    expr: bgpPeerStateNumeric{bgpPeerRemoteAddr!~"^10[.].*",bgpPeerRemoteAddr!~"^172[.].*",role!~"vpn|cfabt1|cfabt2|cfabt3"}
      != 6 unless on(device, bgpPeerRemoteAddr) (ALERTS{alertname=~"BGP_PEER_DOWN_.*"})
    for: 60s
    labels:
      group_remote: '{{ $labels.remote_device }}'
      component: NetworkInfra
      item: BGP
      project: NOC
      severity: "4"
    annotations:
      description: 'Search networking confluence for: ''BGP Peer Down''.'
      summary: BGP peering on {{ $labels.device }} for {{ $labels.bgpPeerRemoteAddr
        }} is down

[simonpasquier]

You probably need to upgrade AlertManager to the latest version. See prometheus/alertmanager#1611

[aixeshunter]

My prometheus version is v2.4.3 too, alertmanager in version 0.15.2, and they are worked.

[simonpasquier]

Closing this as it is most probably fixed by upgrading AlertManager to v0.15.3. Feel free to reopen otherwise.