reload of rules doesn't work, removed/updated rules should get removed/updated

[fvigotti]

• prometheus v1.3.1
• rules are loaded from dir using file globbing

when rules are updated, and reload is triggered ( both using http or signals ) new rules are reloaded correctly
BUT..

• if rule file was previously loaded and the rule internal data has been changed: nothing change ( bug - rules never update if filename doesn't change )
• if rule was previously loaded and another file with same rule name is inserted in the new configuration the rule get's overwritten ( which should be the correct behavior )
• if the rule doesn't exist in the new configuration graph prometheus doesn't delete it ( bug - rules never gets deleted ) in this case, alertmanager restart seems to fix the issue ( but rules are loaded by prometheus so why this happens? this seems a design smell or a problem in the communication between the two anyway.. )
  this point seems to be a fault of alertmanager but I suppose that really the whole issue is inside prometheus which is the one who loads the new rules and should also have the responsability to update alertmanager.

now imagine a monitoring with some microservices / nodes that gets added/removed and maybe a CI system which update alerts from a SCM repository, It's very very frustrating having to reload the whole monitoring system ( prometheus) ( which also cause flush/reload databases ) for every rule added ( which could happen very frequently during some devops working session )

Because I think that alerting is one of the major features of a monitoring system this is a serious bug in my opinion (unfortunately I'm not fluent enough in go to PR myself :( )

Thank you,
Francesco

[beorn7]

Can you give a more concrete example? From what you are saying above, I cannot reproduce the problem.

[brian-brazil]

Without further information, we cannot confirm this is an issue.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.