Error: Expanding alert template failed

[dhbarman]

Proposal

Use case. Why is this important?
This error is preventing us from upgrading Prometheus to 2.3.0 from 2.2.1.

Bug Report

What did you do?

Upgraded Prometheus to 2.3.0 which was earlier running 2.2.1. Config file and rules are as used before.

What did you expect to see?

No error or warning messages on the config parsing.

What did you see instead? Under which circumstances?

Seeing warning messages. Note that we have been using the same config file and rules file (used in config file).

Environment

• System information:

  Linux 4.1.12-124.19.7.el7uek.x86_64 x86_64

• Prometheus version:

  prometheus, version 2.3.0 (branch: heads/v2.3.0, revision: 290d717)
  build user: teamcity@tca-hv07-agent039-default
  build date: 20181030-20:19:00
  go version: go1.11.1

• Prometheus configuration file:

groups:
- name: interface_errors.rule.yml
  rules:
  - alert: FC_INTERFACE_INPUT_ERRORS_OVER_THRESHOLD
    expr: increase(ifInErrors{group=~"fc.*", ifAlias=~".+"}[2m]) > 1 and on(device, interface) (ifOperStatusNumeric == 1)
    for: 3m
    labels:
      component: Test Component
      item: Test Item
      project: TestProject
      severity: "2"
      interface: "{{ $labels.device }}:{{ $labels.interface }}"
    annotations:
      description: '{{ $labels.device }}:{{ $labels.interface }} {{ labels.ifAlias }}'
      summary: FC INTERFACE INPUT ERRORS OVER THRESHOLD device {{ $labels.device }} interface
        {{ $labels.interface }}

• Logs:

level=warn ts=2018-10-30T23:44:04.575030231Z caller=alerting.go:220 component="rule manager" alert=FC_INTERFACE_INPUT_ERRORS_OVER_THRESHOLD msg="Expanding a
lert template failed" err="error parsing template __alert_FC_INTERFACE_INPUT_ERRORS_OVER_THRESHOLD: template: __alert_FC_INTERFACE_INPUT_ERRORS_OVE
R_THRESHOLD:1: function \"labels\" not defined" data="unsupported value type"

[brian-brazil]

This is the better error checking added in 2.3, you've missing a $ on the description which means this alert is currently broken.

[dhbarman]

Thanks @brian-brazil . You may close the ticket.