not receiving alert notifications for CLEARED alarms

[luckman212]

• Netdata v1.20.0-326-nightly
• Ubuntu 19.10

I set up Health check Alarm notifications today, and it went by the book. Alerts going to Twilio SMS and MS-Teams. Everything works. The only thing is, I am not getting alert notifications when the alarms are CLEARED. (These are fping monitored hosts).

Is this normal/expected? Some config I have to tweak? I saw some mention of options: no-clear-notification so I scoured my disk for this setting in all *.conf files but only found a single instance of it in /usr/lib/netdata/conf.d/health.d/net.conf under template: 10s_received_packets_storm which doesn't seem related.

Thanks for any tips!

[luckman212]

update: the CLEARED alert eventually came through at 1:00pm on the dot, which was 20 min after the fact (the actual alarm cleared at 12:40pm). Seems like an hourly cronjob may be rolling these up?

Any way to get them immediately?

[ilyam8]

@luckman212 👋

Probably delay?

https://docs.netdata.cloud/health/reference/#alarm-line-delay

[luckman212]

@ilyam8 That looks promising, will have to set some time aside to read through it, thanks

[ilyam8]

I assume you have delay down option.

[mfundul]

I have problems with alarm-notify.sh which may be related to this. I'm getting:

2020-04-08 21:38:39: alarm-notify.sh: INFO: sent email notification for: CentOS-7-mfundul ipv4.tcphandshake.ipv4_tcphandshake_last_collected_secs is WARNING to 'root'
curl: (3) <url> malformed
2020-04-08 21:38:39: alarm-notify.sh: ERROR: failed to sent CUSTOM_INFO notification for: CentOS-7-mfundul ipv4.tcphandshake.ipv4_tcphandshake_last_collected_secs is WARNING to   with error code 3.

which I've traced back to 3c4cb60

[mfundul]

It doesn't work due to these lines:

# -----------------------------------------------------------------------------
# send the EVENT to Dynatrace
send_dynatrace "${host}" "${chart}" "${name}"  "${status}"
SENT_DYNATRACE=$?

If I comment this one out like this:

#send_dynatrace "${host}" "${chart}" "${name}"  "${status}"

It's fixed.

Is this supposed to be unconditional?

[ilyam8]

I see logic is

check if method is set

netdata/health/notifications/alarm-notify.sh.in

Lines 518 to 523 in 6b63ba8

	# check dynatrace
	{ [ -z "${DYNATRACE_SPACE}" ] ||
	[ -z "${DYNATRACE_SERVER}" ] ||
	[ -z "${DYNATRACE_TOKEN}" ] ||
	[ -z "${DYNATRACE_TAG_VALUE}" ] ||
	[ -z "${DYNATRACE_EVENT}" ]; } && SEND_DYNATRACE="NO"

call the func

netdata/health/notifications/alarm-notify.sh.in

Lines 2464 to 2466 in 6b63ba8

	# send the EVENT to Dynatrace
	send_dynatrace "${host}" "${chart}" "${name}" "${status}"
	SENT_DYNATRACE=$?

Return immediately in func if method is disabled (missing part, no return - added in #8654)

netdata/health/notifications/alarm-notify.sh.in

Lines 1915 to 1921 in 6b63ba8

	# Dynatrace sender
	send_dynatrace() {
	local dynatrace_url="${DYNATRACE_SERVER}/e/${DYNATRACE_SPACE}/api/v1/events"
	local description="NetData Notification for: ${host} ${chart}.${name} is ${status}"
	local payload=""
	payload=$(cat <<EOF

[olivluca]

It appears that the delay down only affects the notification but not the alarm log in the dashboard. Shouldn't the delay also affect what's shown on the dashboard?

[thiagoftsm]

Hi @olivluca ,

What is the Netdata version running on your computer?

best regards!

[olivluca]

v1.21.1

[thiagoftsm]

Hi @olivluca ,

I was testing now in my environment, and I observed that I can receive the clean alarms:

Please, could you give me the alarm that is not sending clear notification for you?

best regards!

[olivluca]

I receive them too but there's a discrepancy between the dashboard (where the alarm shows as cleared immediately) and the clear notification (which is sent only after the delay down time has passed)

[olivluca]

E.g: this alarm (fping.ftpserver_quality) shows as cleared at 9:01 on the dashboard, but the notification was sent at 9:31