Recording Rules take more time in Prometheus 2.3.0 after upgrade

[dhbarman]

Proposal

Use case. Why is this important?

Prometheus recording rules should not take more time

Bug Report

What did you do?

Upgraded Prometheus 2.2.1 to 2.3.0. The same set of recording rules were taking less time with previous version.

What did you expect to see?

Alarms based on metrics (recording rule evaluation) should not have gotten fired.

What did you see instead? Under which circumstances?

We are observing alarms firing.

Environment

• System information:

  Linux 4.1.12-124.19.7.el7uek.x86_64 x86_64

• Prometheus version:

  prometheus, version 2.3.0 (branch: heads/v2.3.0, revision:
  290d717)
  build user: teamcity@tca-hv05-agent018-default
  build date: 20181030-04:34:03
  go version: go1.11.1

[simonpasquier]

Can you share the recording rules and the definition of the firing alerts?