Prometheus continues to scrape removed targets from empty scrape configs

[grobie]

Bug Report

What did you do?

Removed last entry of a static_configs directive. Our template generator omits empty config blocks from being written out to disk. The written out scrape configuration didn't have any *_sd_configs or static_configs configuration block.

What did you expect to see?

Either Prometheus rejects scrape configurations without a single discovery config (neither static nor any *_sd_configs), or it handles such empty scrape configurations correctly.

What did you see instead? Under which circumstances?

Prometheus didn't show any targets in the targets page, but continued to scrape the target from the last static_configs (with up == 0 as the target was deprovisioned).

Environment

• System information:

  Linux 4.18

• Prometheus version:

  2.4.2

@krasi-georgiev

[simonpasquier]

I'll have a look at it.

[simonpasquier]

Hmm it seems even worse with master (and the future 2.5) as it doesn't even remove the scrape configuration from the targets page when the last target is removed. It isn't "worse", it is just that the bug is less hidden.

Edit: same behavior with 2.3 too.