When many hosts go offline, Recache Event can be constantly logged

[bmfmancini]

I have found that when you have multiple devices go offline say after a network hit that a recache loop gets triggered

2022-08-17 15:38:00 - PCOMMAND Device[1105] NOTE: Recache Event Detected for Device
2022-08-17 15:37:50 - PCOMMAND Device[1102] NOTE: Recache Event Detected for Device
2022-08-17 15:37:42 - PCOMMAND Device[1100] NOTE: Recache Event Detected for Device
2022-08-17 15:37:34 - PCOMMAND Device[1099] NOTE: Recache Event Detected for Device
2022-08-17 15:37:26 - PCOMMAND Device[1097] NOTE: Recache Event Detected for Device
2022-08-17 15:37:19 - PCOMMAND Device[1096] NOTE: Recache Event Detected for Device
2022-08-17 15:37:11 - PCOMMAND Device[1089] NOTE: Recache Event Detected for Device
2022-08-17 15:37:03 - PCOMMAND Device[1087] NOTE: Recache Event Detected for Device

in this case these devices have a large amount of OID's being pulled so they take time to respond
the poller_commands.php script times out even after 20 minutes due to how long it take to run through each device

However I notice that the poller_commands table does not update even after your reindex the devices manually
I also switched all of the reindex methods to none and still the poller_commands script runs

Example device 1576

2022-08-17 15:26:38 - PCOMMAND Device[1576] NOTE: Recache Event Detected for Device
2022-08-17 14:46:50 - PCOMMAND Device[1576] NOTE: Recache Event Detected for Device

manually re-indexing the device completes without error however the poller_command table still has an entry for that device

MariaDB [cacti]> select * from poller_command where command like '%1576%'  \G
*************************** 1. row ***************************
   poller_id: 1
        time: 2022-07-20 20:40:02
      action: 1
     command: 1576:10
last_updated: 2022-07-20 20:40:02
*************************** 2. row ***************************
   poller_id: 1
        time: 2022-07-20 20:40:02
      action: 1
     command: 1576:4
last_updated: 2022-07-20 20:40:02
2 rows in set (0.001 sec)

reindex method is set to none for this device yet poller_commands still has it set for recache

MariaDB [cacti]> select * from host_snmp_query where host_id = 1576;
+---------+---------------+------------+-----------------+----------------+
| host_id | snmp_query_id | sort_field | title_format    | reindex_method |
+---------+---------------+------------+-----------------+----------------+
|    1576 |             4 | ifIndex    | |query_ifIndex| |              0 |
|    1576 |            10 | t1         | |query_t1|      |              0 |
+---------+---------------+------------+-----------------+----------------+
2 rows in set (0.000 sec)

Spine reports the device has nothing for recache

2022-08-17 15:50:19 - SPINE: Poller[1] PID[3914359] PT[140412896401152] Device[1576] HT[1] Device has no information for recache.
2022-08-17 15:50:19 - SPINE: Poller[1] PID[3914359] PT[140412896401152] Device[1576] HT[1] NOTE: There are '78' Polling Items for this Device

[bmfmancini]

Workaround for this

I truncated the poller_command table make sure poller_commands is not running
if it is killing the process or you may still see recache events