Deleted rules don't go stale after a reload

[jsravn]

Bug Report

What did you do?

I noticed this while digging into #5193.

If you delete a recording rule from the end of your rules and reload, the rule doesn't go stale.

What did you expect to see?

After deleting a rule and reloading, I expect it to go stale in the next eval cycle.

What did you see instead? Under which circumstances?

The recording rule time series sticks around for 5m.

Environment

docker container

• System information:

/prometheus $ uname -srm
Linux 4.20.6-arch1-1-ARCH x86_64

• Prometheus version:

/prometheus $ prometheus --version
prometheus, version 2.7.1 (branch: HEAD, revision: 62e591f928ddf6b3468308b7ac1de1c63aa7fcf3)
  build user:       root@f9f82868fc43
  build date:       20190131-11:16:59
  go version:       go1.11.5

• Prometheus configuration file:

/etc/prometheus $ cat prometheus.yml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.                             
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.                                     
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.                                  
rule_files:                
- "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:                                                              
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.                                
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

/etc/prometheus $ cat first_rules.yml 
groups:
- name: testgroup
  rules:
  - record: test
    expr: 11111
    labels:
      name: my-first
  - record: test
    expr: 22222
    labels:
      name: my-second

[simonpasquier]

Thanks for reporting it, I've stumbled upon the same issue while investigating #5193 too 😄
AFAICT both issues can be fixed in the same row.