Add support for enable/disable some checks

[lyrixx]

I have lot of checks in my DC, but I don't want every check triggers an alert. Some check are not so important (ram usage, load usage). they are here just for quick overview of our DC, and to quickly find an error when something goes wrong.

So, I see at least two options to configure this:

1. Add a black-list of check name consul-alerts/config/checks/black-list : ["load_usage", "memory_usage"].
2. By check on node: consul-alerts/checks/consumer-0.insight-d1/_/disk_usage_disable: true

I think the first option better fits my use case, because I want to ignore (all load|memory)_usage. But the second one is more granular.

What do you think?

[bcwilsondotcom]

+1 on this. One of my concerns with moving all of my monitoring/alerting from Sensu to Consul is not easily being able to enable/disable alerting on checks.

I think both of lyrixx's options are valid scenarios. Either I have a node which I want to disable alerting on temporarily, or, I have a check script which is behaving undesirably on all nodes and I want to disable that check completely.

[hunter]

Although it would add complexity, it might actually be useful to allow both a full black list and individual hosts (or even tags).

[darkcrux]

I like both idea. We can possibly add both. The blacklist would have more weight than the individual config.

Basically, if it's blacklisted, ignore it right away. If not, check if it's enabled/disabled. Tags would probably be the most flexible (and more complex) solution.

[darkcrux]

Hi guys, just release v0.1.2 with a blacklist feature. Config guide is in the README. Check it out. I think we've covered most use-cases with this. We can implement a tag based solution sometime soon.

[lyrixx]

Huhu, I just noticed it in my logs

time="2014-11-14T09:59:15Z" level="info" msg="Unable to check blacklist for frontend-1.insight-d1:_:database, return false"

Thanks, I will be able to "stop" checks during deploy, and to always discard some checks ;)

[darkcrux]

There's two reason for that message (it should have been a warning, not an info, will fix the logs). Either consul-alerts/config/checks/frontend-1.insight-d1/_/database/blacklisted does not exist, or the value is not true or false. Or maybe a typo? ;)

[lyrixx]

Yes, the value does not exist (yet). I just noticed that when I tried to debbuging the issue with slack. ;)

[darkcrux]

haha. yeah. Logging needs better cleanup. I'll add it to my list.

[darkcrux]

Just release v0.2.0 with changes to how this is implemented. Fixed a possible race condition which @lyrixx pointed out. It's much simpler now.

Check out the README for more info. Cheers.

[lyrixx]

perfect ;)