Recovery threshold email not sent

[k4y53r]

Hi,

I've detect that some recovery email from graph templates are not sending from Cacti, up/down hosts are sent fine

Thold version -> 1.0.4
Cacti version -> 1.1.35

I've checked logs and no line from MAILER generated to send recovery email, only breached mail logged
On thold log i could see event for threshold triggered and restored

It fails with TCP port and avaliable used space graph templates at least

After put plugin in developer mode i could locate more info on logs as you could check below, also you could see thold records on cacti database with both events (triggered and restored);

-- phpMyAdmin SQL Dump
-- version 4.5.4.1deb2ubuntu2

-- Servidor: localhost
-- Tiempo de generación: 06-03-2018 a las 12:40:48
-- Versión del servidor: 5.7.21-0ubuntu0.16.04.1
-- Versión de PHP: 7.0.25-0ubuntu0.16.04.1

SET SQL_MODE = "NO_AUTO_VALUE_ON_ZERO";
SET time_zone = "+00:00";

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT /;
/!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS /;
/!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION /;
/!40101 SET NAMES utf8mb4 */;

-- Estructura de tabla para la tabla plugin_thold_log

CREATE TABLE plugin_thold_log (
id int(12) NOT NULL,
time int(24) NOT NULL,
host_id int(10) NOT NULL,
local_graph_id int(10) NOT NULL,
threshold_id int(10) NOT NULL,
threshold_value varchar(64) NOT NULL,
current varchar(64) NOT NULL,
status int(5) NOT NULL,
type int(5) NOT NULL,
description varchar(255) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COMMENT='Table of All Threshold Breaches';

--
-- Volcado de datos para la tabla plugin_thold_log

INSERT INTO plugin_thold_log (id, time, host_id, local_graph_id, threshold_id, threshold_value, current, status, type, description) VALUES
...

(983, 1520339559, 17, 455, 1114, '1', '0', 4, 0, 'ALERT: HOSTNAME - TCP State - 80 [TCP] went below threshold of 1 with 0'),
(984, 1520339859, 17, 455, 1114, '', '1', 5, 0, 'NORMAL: HOSTNAME - TCP State - 80 [TCP] Restored to Normal Threshold with Value 1');

Logs on developer mode for thold plugin:

2018/03/06 12:32:39 - MAILER Mail Sucessfully Sent to '<EMAILS_FROM_NOTIFICATION_LIST>', Subject: 'ALERT: HOSTNAME - TCP State - 80 [TCP] went below threshold of 1 with 0'
2018/03/06 12:37:39 - THOLD: Threshold Breached ID: 1114 DataTemplate: TCP DataSource: TCP Type: High / Low Enabled: on Current: 1 High: Low: 1 Trigger: 2 Warning High: Warning Low: Warning Trigger: 0 SentTo:

As you coul see on previous line no emails are listed on "SentTo:" field.

[netniV]

Can you check:
Console -> Configuration -> Settings -> Threshold -> SNMP Notification Presets -> Disable Restoration Notifications

If that is ticked, you will not get notifications.

[k4y53r]

Hi,

I've checked that and it's not checked.

As you could see below problem is that no recipient is set on destination email:
THOLD:
Threshold Breached ID: 1114
DataTemplate: TCP
DataSource: TCP
Type: High / Low
Enabled: on
Current: 1
High:
Low: 1
Trigger: 2
Warning High:
Warning Low:
Warning Trigger: 0
SentTo:

This thold alert when TCP port value are below 1, and value shown above are 1 so it should send restoration email but "SendTo:" field are empty

Device UP/DOWN for this device ID works fine, only fails recovery thresholds for some graphs. Alert threshold emails works fine

Graphs tholds not working detected:

• TCP
• Advanced Ping [loss]
• HDD Used

[netniV]

What are your alert/warning settings? Can you provide a screen shot of them all?

[k4y53r]

Hi,

You could see fresh example from this weekend with logs and screenshots of thold breached but no recovery mail sent from cacti (only breached)

-- Log breached and restored shown on thold log:

-- Only breached mail shown on Cacti Log:

-- Thold config for this threshold:

[netniV]

OK, so your settings look OK. I'll take a look later to see if I can spot anything in the code. I'm in routerconfigs mode right now so don't want to get my head mixed up ;-)

[k4y53r]

No problem, i could not locate any pattern, sometimes fails tcp thold, others hdd used, ping loss and it happens on almost 3 cacti installations with same thold version

[netniV]

Do you have the latest develop branch?

[k4y53r]

Same version number, but installed maybe a month ago...

[netniV]

I have been reviewing the code as part of another issue and I think there should be a log that email notification was suppressed if there was a reason to suppress it. That may help in this situation, or it might not. There is also some work going to be done to verify the whole breach/notification system to document what we believe the process should be doing, and what it's actually doing.

[k4y53r]

Thank you so much

I'll wait till this work is done

Regards...

[TheWitness]

I would suggest that you test again using the latest stable development. There have been a number of bug fixes and this may have been included. More importantly, there have been several mailer issue fixes since 1.1.35 around ssl and non ssl communications. Note that the lastest development is calling for 1.2.1, though you can mostly use it at 1.2.0 if you change the INFO file.

[k4y53r]

Ok, i'll try in a couples of days on test environment with v 1.2.1

Thanks and regards....