An alert is not fired again after it has been resolved once

[alon-argus]

It looks like a critical bug, in the new 0.17.0rc1 pre-release version.

Reproduction steps:

1. Make some alert to fire (Alertmanager gets an alert)
2. Resolve it so it'll stop firing (Alertmanager alert is cleared)
3. Make the alert to fire again

Expected result:
Alert should appear as 'firing' in 'Alerts' page, and sent to Alertmanager.

Actual result:
Alert is not firing in 'Alerts' page ((0 active)), and Alertmanager doesn't get it as well.

Additional information:
My alert:

ALERT vehicleServer_response_err
  IF (sum(rate(server_requests_processing_time_millis_total{status!="200"}[15s])) BY (status)) > 0
  FOR 40s
  ANNOTATIONS {
    description="server returns non-200 status code, for more than 40 seconds (status:  {$labels.status}}).",
    summary="server returned non-200 status code"
  }

When the alert's rule triggers again from a different status code, the alert does fire though.

Can't see any errors in stdout.

[grobie]

I can reproduce.

# prometheus.yml
global:
  scrape_interval:     5s
  evaluation_interval: 5s

rule_files:
  - 1372.rules

scrape_configs:
  - job_name: 'node_exporter'
    target_groups:
      - targets: ['localhost:9100']

# 1372.rules
ALERT vehicleServer_response_err
  IF foobar{status!="200"} > 0
  FOR 40s
  ANNOTATIONS {
    description="server returns non-200 status code, for more than 40 seconds (status:  {$labels.status}}).",
    summary="server returned non-200 status code"
  }

./prometheus &
./node_exporter --collector.textfile.directory=data/ &

while true; do
  echo "foobar{status=\"200\"} 42\nfoobar{status=\"500\"} 0" > data/1372.prom
  sleep 60
  echo "foobar{status=\"200\"} 2\nfoobar{status=\"500\"} 100" > data/1372.prom
  sleep 60
done

[grobie]

Note this only affects flapping alerts with a flap cycle of <15m.

[fabxc]

Thanks a lot for catching that one!

[alon-argus]

Cool!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.