Alert only based on specific "string" in a text file.

[ridhoswasta]

Is it possible to have a lookup or warn/crit based on a string inside a file?
We have a MASTER-BACKUP condition, so we only want to monitor services when the server is in a master state.

alarm: maptool_processes
on: apps.processes
lookup: min -5s match-names of maptool_tassta
every: 5s
crit: $this != nan AND $this = 0 AND cat /usr/serverstate = MASTER
units: processes
info: maptool on master server currently is out of process [down / fail ]

Thanks!

[andrewm4894]

I think you could make a copy of the alarm-notify.sh script that maybe checks what you want from the file and just breaks if condition not met maybe: https://learn.netdata.cloud/docs/agent/health/notifications

@cakrit i recall a user asking something similar a while ago in the community i think - can you remember?

Another option could be to make a collector to check the value of the file at regular time intervals and map the file contents to a state number like 0,1,2 etc and then you'd maybe be able to create 2 alert configs - one for the app.processes like above and another depending on the state of the number 0,1,2.

This example shows combining two alerts, so thinking maybe something similar: https://learn.netdata.cloud/docs/agent/health/reference#example-7---z-score-based-alarm

I had a look but dont think the filecheck collector can actually read contents of the file; https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/filecheck

But if you could have it be something like file /usr/serverstate/master and file /usr/serverstate/notmaster existence or not determines the state then you could maybe even just use the filecheck collector to determine the state and do the combine two alerts approach i mention.

[ridhoswasta]

Thank you so much for your reply, it's giving me some Idea. I wanted to apply your first suggestion by modifying alarm-notify.sh but I am not sure where to put this check and break it from being sent as alarm.

[cakrit]

You would need to provide a custom notification method and configure health_alarm_notify.conf to send those custom notifications.

Another option is to use the Health Management API to enable/disable any check or alert notification you want. So instead of checking the contents of a file when an alert gets triggered you could link the API commands it to whatever triggers the state changes themselves, to completely stop the checks themselves. If it's not possible to hook into the cause of the state changes directly, you could perhaps watch the file with https://github.com/inotify-tools/inotify-tools/wiki#inotifywait or maybe even watch (but I wouldn't personally like to depend on a watcher like this, you'd need to ensure its always running, which can be tricky.)