Alert labels are not changed after update rule file and reloading prometheus.

[krasoffski]

Bug Report

1. Download prometheus version v2.0.0 and v2.3.2 (reproduced on both).
2. Extract these version to /tmp directory

tmp $ ls ./prometheus*
./prometheus-2.0.0.linux-amd64.tar.gz  ./prometheus-2.3.2.linux-amd64.tar.gz

./prometheus-2.0.0.linux-amd64:
console_libraries  consoles  LICENSE  NOTICE  prometheus  prometheus.yml  promtool

./prometheus-2.3.2.linux-amd64:
console_libraries  consoles  LICENSE  NOTICE  prometheus  prometheus.yml  promtool

3. Create new prometheus configuration file /tmp/prometheus.yml with following content:

global:
  scrape_interval:     30s
  evaluation_interval: 5s
  scrape_timeout:      5s

alerting:
  alertmanagers:
  - path_prefix: /reciver
    scheme: http
    static_configs:
    - targets:
      - 127.0.0.1:19999
    timeout: 10s

rule_files:
  - /tmp/rules/*.yaml

scrape_configs:

  - job_name: "Test metric"
    metrics_path: /
    honor_labels: true
    scrape_timeout: 30s
    static_configs:
      - targets:
        - 127.0.0.1:8080

4. Start simple metric generator like bellow:

#!/usr/bin/env python2

from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import SocketServer
import random

class Server(BaseHTTPRequestHandler):
    def _set_headers(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/plain')
        self.end_headers()

    def do_GET(self):
        self._set_headers()
        self.wfile.write('my_awesome_metric{app="foobar"} ' + str(random.randrange(5,10)))


def run(server_class=HTTPServer, handler_class=Server, port=8080):
    server_address = ('', port)
    httpd = server_class(server_address, handler_class)
    print 'Starting...'
    httpd.serve_forever()

if __name__ == "__main__":
    from sys import argv

    if len(argv) == 2:
        run(port=int(argv[1]))
    else:
        run()

5. Start dummy alert manager like bellow (you can use nc linux utility for this):

package main

import (
	"flag"
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
)

func handler(w http.ResponseWriter, r *http.Request) {
	_, err := io.Copy(os.Stdout, r.Body)
	defer r.Body.Close()
	if err != nil {
		panic(err)
	}
	fmt.Println("\n")
	w.WriteHeader(http.StatusOK)
}

func main() {
	port := flag.String("port", "19999", "port to listen on")
	path := flag.String("path", "/reciver/api/v1/alerts", "path to listen on")
	addr := fmt.Sprintf("0.0.0.0:%s", *port)
	log.Printf("Starting server %s%s", addr, *path)
	http.HandleFunc(*path, handler)
	log.Fatal(http.ListenAndServe(addr, nil))

6. Create directory /tmp/rule1 and place alert rule template and also place this rule to rule_files
   directory with path /tmp/rules/ (from prometheus configuration file).

groups:
- name: my_group
  interval: 5s
  rules:
  - alert: my_alert
    expr: my_awesome_metric > 1
    for: 5s
    labels:
      rule_prefix: my.rule.prefix
      rule1_lavel: rule1_value

cp /tmp/rule1/rule.yaml /tmp/rules/rule.yaml

7. Create directory /tmp/rule2 and place alert rule template (notice different labels)

groups:
- name: my_group
  interval: 5s
  rules:
  - alert: my_alert
    expr: my_awesome_metric > 1
    for: 5s
    labels:
      rule_prefix: my.rule.prefix
      rule2_lavel: rule2_value

8. Start Prometheus with command

tmp $ ./prometheus-2.0.0.linux-amd64/prometheus --config.file=./prometheus.yml --web.listen-address=0.0.0.0:9100 --web.enable-lifecycle

9. Wait till new alerts appears in the output of dummy manager:

tmp $ ./dummy-manager 
2018/09/03 17:48:56 Starting server 0.0.0.0:19999/reciver/api/v1/alerts
[{"labels":{"alertname":"my_alert","app":"foobar","instance":"127.0.0.1:8080","job":"Test metric","rule1_lavel":"rule1_value","rule_prefix":"my.rule.prefix"},"annotations":{},"startsAt":"2018-09-03T17:49:27.076313109+03:00","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://BY-LL-022:9100/graph?g0.expr=my_awesome_metric+%3E+1\u0026g0.tab=1"}]

[{"labels":{"alertname":"my_alert","app":"foobar","instance":"127.0.0.1:8080","job":"Test metric","rule1_lavel":"rule1_value","rule_prefix":"my.rule.prefix"},"annotations":{},"startsAt":"2018-09-03T17:49:27.076313109+03:00","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://BY-LL-022:9100/graph?g0.expr=my_awesome_metric+%3E+1\u0026g0.tab=1"}]

10. Overwrite rule.yaml within /tmp/rules directory by rule from /tmp/rule2/rule.yaml, see step 7. This rule has the same group and alert names but different labels. Perform reload with command:

$ cp /tmp/rule2/rule.yaml /tmp/rules/rule.yaml
$ curl -X POST http://localhost:9100/-/reload

What did you expect to see?
I expect to see json alert which contains new labels. UI shows correct labels but prometheus sends alert with old labels to alert manager.

What did you see instead? Under which circumstances?
I see alert with old labels instead of new ones which defined in the /tmp/rule2/rule.yaml file.

Environment

• System information:

  uname -srm
  Linux 4.15.0-33-generic x86_64
  

• Prometheus version:

tmp $ ./prometheus-2.0.0.linux-amd64/prometheus --version
prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)
  build user:       root@615b82cb36b6
  build date:       20171108-07:11:59
  go version:       go1.9.2
tmp $ ./prometheus-2.3.2.linux-amd64/prometheus --version
prometheus, version 2.3.2 (branch: HEAD, revision: 71af5e29e815795e9dd14742ee7725682fa14b7b)
  build user:       root@5258e0bd9cc1
  build date:       20180712-14:02:52
  go version:       go1.10.3

• Alertmanager version:

  Did not use.

• Prometheus configuration file:
  Please, see above.

[brian-brazil]

This sounds like #4169. Can you try at master?

[krasoffski]

@brian-brazil, thank you for quick replay.
Will try to reproduce today/tomorrow using the latest master and post results here.

[krasoffski]

@brian-brazil am not able to get working configuration using Prometheus from master now (using the same configs and setups).
It seems like Prometheus does not see/use targets and alert managers from prometheus.yml configuration file (no targets and alert managers avalibale via WEB UI, no request to targets).
It is possible workaround with absent missing targets, but it would be great to have JSON alert sent to dummy alert manager.

BTW, will try to use build based on issue fix mentioned above (commit/cherry-pick instead of master).

[simonpasquier]

@krasoffski you're probably hitting #4551 (in short master is currently broken for SD). #4556 should fix it so you may want to build from there.

[simonpasquier]

Can you retest with the latest v2.4.2 release as it should be fixed now?

[krasoffski]

@simonpasquier thank you. I will try to retest issue and re-open defect in case of issue is still valid.

[simonpasquier]

Thanks @krasoffski