Undefined index: timeout_exceed /lib/poller.php on line: 1953

[jdcoats]

2021/10/22 10:35:18 - POLLER: Poller[Main Poller] PID[8312] ERROR: Process being killed due to timeout! (commands, master, 1, Process 23662, Time , Timeout 300, Timestamp 1634913318)
2021/10/22 10:35:18 - CMDPHP PHP ERROR NOTICE Backtrace: (/poller_commands.php[121]:register_process_start(), /lib/poller.php[1953]:CactiErrorHandler())
2021/10/22 10:35:18 - ERROR PHP NOTICE: Undefined index: timeout_exceed in file: /var/www/localhost/htdocs/cacti/lib/poller.php on line: 1953

[jdcoats]

I'm also stuck with constant recaching of 100+ devices.

NOTE: Recache Event Detected for Device
RECACHE ASSERT FAILED:
 

[TheWitness]

Please test and close if resolved. Thanks!

[jdcoats]

I will as soon as I can recover whatever is going on. I updated Cacti & Spine this morning and was getting recache events constantly for many devices. I ended up rolling back to yesterdays Cacti and Spine from Oct 1. I'm missing many datasources now. Still trying to figure out how to calm the storm.

2021/10/22 11:52:01 - POLLER: Poller[Main Poller] PID[22424] WARNING: Poller Output Table not Empty. Issues: 98, DS[Missing Datasource 71240, Missing Datasource 71245, Missing Datasource 71225, Missing Datasource 71152, Missing Datasource 71310, Missing Datasource 71285, Missing Datasource 71275, Missing Datasource 71260, Missing Datasource 71305, Missing Datasource 71340, Missing Datasource 71290, Missing Datasource 71132, Missing Datasource 71325, Missing Datasource 71196, Missing Datasource 71142, Missing Datasource 71112, Missing Datasource 71235, Missing Datasource 71270], Additional Issues Remain. Only showing first 20

[jdcoats]

for example 100 PDU's without snmp interface stats on the device or graphs started doing this:

2021/10/22 11:58:01 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-4052-PDU-02] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
--
2021/10/22 11:57:59 - PCOMMAND Device[C-Mustian-4052-PDU-02] NOTE: Recache Event Detected for Device
2021/10/22 11:57:59 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-4052-PDU-01] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:57 - PCOMMAND Device[C-Mustian-4052-PDU-01] NOTE: Recache Event Detected for Device
2021/10/22 11:57:57 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-4033-PDU-03] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:56 - PCOMMAND Device[C-Mustian-4033-PDU-03] NOTE: Recache Event Detected for Device
2021/10/22 11:57:56 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-4033-PDU-02] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:54 - PCOMMAND Device[C-Mustian-4033-PDU-02] NOTE: Recache Event Detected for Device
2021/10/22 11:57:53 - PCOMMAND Device[C-Mustian-4033-PDU-01] NOTE: Recache Event Detected for Device
2021/10/22 11:57:53 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-3052-PDU-03] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:51 - PCOMMAND Device[C-Mustian-3052-PDU-03] NOTE: Recache Event Detected for Device
2021/10/22 11:57:51 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-3052-PDU-02] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:49 - PCOMMAND Device[C-Mustian-3052-PDU-02] NOTE: Recache Event Detected for Device
2021/10/22 11:57:49 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-3052-PDU-01] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:47 - PCOMMAND Device[C-Mustian-3052-PDU-01] NOTE: Recache Event Detected for Device
2021/10/22 11:57:47 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-3033-PDU-03] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:45 - PCOMMAND Device[C-Mustian-3033-PDU-03] NOTE: Recache Event Detected for Device
2021/10/22 11:57:45 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-3033-PDU-02] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:43 - PCOMMAND Device[C-Mustian-3033-PDU-02] NOTE: Recache Event Detected for Device
2021/10/22 11:57:42 - PCOMMAND Device[C-1POB-4-SW48-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 11:57:42 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-3033-PDU-01] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:41 - PCOMMAND Device[C-Mustian-3033-PDU-01] NOTE: Recache Event Detected for Device
2021/10/22 11:57:41 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-1470-PDU-03] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:40 - PCOMMAND Device[C-Mustian-1470-PDU-03] NOTE: Recache Event Detected for Device
2021/10/22 11:57:40 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-1470-PDU-02] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:39 - PCOMMAND Device[C-Mustian-1470-PDU-02] NOTE: Recache Event Detected for Device
2021/10/22 11:57:39 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-1470-PDU-01] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:38 - PCOMMAND Device[C-Mustian-1470-PDU-01] NOTE: Recache Event Detected for Device
2021/10/22 11:57:38 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-1459-PDU-03] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:36 - PCOMMAND Device[C-Mustian-1459-PDU-03] NOTE: Recache Event Detected for Device

[TheWitness]

Well, there was a broken spine a few days ago around constant recaches. It was fixed a day or so ago. So, if you update spine, make sure you get the latest. Also the latest spine depends on the column snmp_sysUpTimeInstance in the host table being bigint unsigned. You might have to manually update that column.

[TheWitness]

The flipping of ifName to ifDescr is almost like you have modified the resource XML file to use ifDescr as the sort key. I hope that was not a part of the package. Checking from my build machine.

[TheWitness]

The packaged version of interface.xml is correct.

[jdcoats]

using spine built this morning. using the default packages, even reinstalled all of them. I did modify int to bigint

ALTER TABLE host MODIFY COLUMN snmp_sysUpTimeInstance bigint(20) unsigned not null default "0";
ALTER TABLE automation_devices MODIFY COLUMN sysUpTime bigint(20) unsigned not null default "0";

problem persists

[jdcoats]

also still getting that process killed due to timeout but without the undefined index :)

2021/10/22 17:16:19 - PCOMMAND Device[M-Bradfordville-SW48-01] NOTE: Recache Event Detected for Device
2021/10/22 17:16:17 - PCOMMAND Device[M-Blountstown-Rehab-SW24-01] NOTE: Recache Event Detected for Device
2021/10/22 17:16:17 - POLLER: Poller[Main Poller] PID[5170] ERROR: Process being killed due to timeout! (commands, master, 1, Process 20129, Time 1634937020, Timeout 300, Timestamp 1634937377)
2021/10/22 17:16:17 - SYSTEM STATS: Time:16.3599 Method:spine Processes:2 Threads:40 Hosts:1774 HostsPerProcess:887 DataSources:121798 RRDsProcessed:0
2021/10/22 17:16:14 - PCOMMAND Device[H-1N-36-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:16:11 - PCOMMAND Device[H-1N-33-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:16:07 - PCOMMAND Device[H-GS-152-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:16:05 - PCOMMAND Device[H-CompRoom-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:16:02 - PCOMMAND Device[H-GN-50-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:16:00 - PCOMMAND Device[H-Facilities-STCK-01] NOTE: Recache Event Detected for Device
2021/10/22 17:15:59 - PCOMMAND Device[M-Carrabelle-SW24-01] NOTE: Recache Event Detected for Device
2021/10/22 17:15:58 - PCOMMAND Device[M-Apalachicola-SW24-01] NOTE: Recache Event Detected for Device
2021/10/22 17:15:55 - PCOMMAND Device[H-GB-41-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:15:51 - PCOMMAND Device[H-GA-38-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:15:49 - PCOMMAND Device[H-GA-2-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:15:47 - PCOMMAND Device[H-MDF-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:15:45 - PCOMMAND Device[H-BN-65-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:15:44 - PCOMMAND Device[M-FSU-SW24-01] NOTE: Recache Event Detected for Device
2021/10/22 17:15:41 - PCOMMAND Device[M-Heart-Rhythm-STCK-01] NOTE: Recache Event Detected for Device