{{$vaule}} form alert rule not correct

[mozhata]

What did you do?

I write an alert rule something like

ALERT container_network_receive_bytes_total_lt_102400
  IF (ceil(sum(rate(container_network_receive_bytes_total{id="/",instance="ubuntu-25",interface=~"em.*|eth.*|en.*"}[5m])) / 1024 * 100) / 100) < 100
  LABELS {tenxTargetName="ubuntu-25", tenxTargetType="node"}
  ANNOTATIONS { currentValue="{{ $value }}KB/s"}

and some log at alertmanager API POST /api/v1/alerts to check the {{ $value }}'s real value:

func (api *API) insertAlerts(w http.ResponseWriter, r *http.Request, alerts ...*types.Alert) {
	now := time.Now()

	for _, alert := range alerts {
		for ln, lv := range alert.Labels {
			if lv == "ubuntu-25" {
				fmt.Printf("\n\nreceived data form api: ln: %s lv: %s\nalert: %s", ln, lv, handy.MarshalJSONOrDie(alert))
			}
		}
		alert.UpdatedAt = now

		// Ensure StartsAt is set.
		if alert.StartsAt.IsZero() {
			alert.StartsAt = now
		}
...

What did you expect to see?

the {{ $value }}'s real value should changed sometime
What did you see instead? Under which circumstances?
I found every time the log out is just fixed value like 2.3KB/s and never changed
and far form value the got by promethe webUI

Environment

• System information:

  Linux 4.4.0-31-generic x86_64

• Prometheus version:

prometheus, version 1.0.1 (branch: master, revision: be40190)
  build user:       root@e881b289ce76
  build date:       20160722-19:54:46
  go version:       go1.6.2

• Alertmanager version:

alertmanager, version 0.5.1 (branch: HEAD, revision: 0ea1cac51e6a620ec09d053f0484b97932b5c902)
  build user:       kang@kang-pc
  build date:       20170426-09:35:23
  go version:       go1.7.4

• Prometheus configuration file:

global:
  scrape_interval: 20s
  scrape_timeout: 20s
rule_files:
- /etc/prometheus_rules/*.rule
scrape_configs:
- job_name: 'prometheus'
  static_configs:
    - targets: ['localhost:9090','192.168.1.24:9100','192.168.1.25:9100']
- job_name: 'kubernetes-cluster'
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
  - api_servers:
    - 'https://10.0.0.1:443'
    in_cluster: true
    role: apiserver
- job_name: 'kubernetes-nodes'
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
  - api_servers:
    - 'https://10.0.0.1:443'
    in_cluster: true
    role: node
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
- job_name: 'kubernetes-service-endpoints'
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
  - api_servers:
    - 'https://10.0.0.1:443'
    in_cluster: true
    role: endpoint
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: (.+)(?::\d+);(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_service_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name
- job_name: 'kubernetes-services'
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  metrics_path: /probe
  params:
    module: [http_2xx]
  kubernetes_sd_configs:
  - api_servers:
    - 'https://10.0.0.1:443'
    in_cluster: true
    role: service
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
    action: keep
    regex: true
  - source_labels: [__address__]
    target_label: __param_target
  - target_label: __address__
    replacement: blackbox
  - source_labels: [__param_target]
    target_label: instance
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_service_namespace]
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    target_label: kubernetes_name
- job_name: 'kubernetes-pods'
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
  - api_servers:
    - 'https://10.0.0.1:443'
    in_cluster: true
    role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: (.+):(?:\d+);(\d+)
    replacement: ${1}:${2}
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_pod_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name/prometheus 

• Alertmanager configuration file:

{
    "global": {
        "resolve_timeout": "30s",
        "smtp_smarthost": "localhost:25",
        "smtp_from": "alertmanager@example.org",
        "smtp_auth_username": "alertmanager",
        "smtp_auth_password": "password"
    },
    "route": {
        "group_by": [
            "tenxNamespace",
            "tenxStrategyName",
            "tenxTargetName",
            "tenxTargetType",
            "tenxClusterID"
        ],
        "group_wait": "5s",
        "group_interval": "5m",
        "repeat_interval": "0s",
        "receiver": "default",
        "routes": [
            {
                "match_re": {
                    "alertname": ".*"
                },
                "continue": true,
                "receiver": "five-minutes",
                "group_interval": "5m"
            },
            {
                "match_re": {
                    "alertname": ".*"
                },
                "continue": true,
                "receiver": "ten-minutes",
                "group_interval": "10m"
            }
        ]
    },
    "receivers": [
        {
            "name": "default"
        },
        {
            "name": "five-minutes",
            "webhook_configs": [
                {
                    "url": "http://192.168.0.65:8000/spi/v2/alerts/notifications/intervals/300",
                    "send_resolved": false
                }
            ]
        }
    ]
}

[brian-brazil]

Can you confirm this with 1.6.1? I believe we fixed this some time ago.

[mozhata]

@brian-brazil
I change the vertion to:\

prometheus, version 1.6.1 (branch: master, revision: 4666df502c0e239ed4aa1d80abbbfb54f61b23c3)
  build user:       root@7e45fa0366a7
  build date:       20170419-14:32:22
  go version:       go1.8.1

alertmanager, version 0.5.1 (branch: HEAD, revision: 0ea1cac51e6a620ec09d053f0484b97932b5c902)
  build user:       kang@kang-pc
  build date:       20170427-08:08:01
  go version:       go1.7.4

and the problem dispeared, thanks !

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.