Polling cycles may not always complete as expected

[nicolatron]

I have a setup with one main poller and 2 additional pollers. With boost enabled.

Upgraded recently from 1.2.18 to 1.2.19, I was running spine 1.2.16, becouse more recent version (1.2.17 or 18) was giving errors, now I'm running both cacti and spine 1.2.19. Apparently spine ends without problems but process "php /opt/cacti/poller.php" is unable to finish in main poller, and keeps there eating memmory and cpu. Ends up being killed after 299 seconds. Weirdly enougth 3 of that process are running at the same time, wich is strage having they are killed after 299 seconds, and cron is set to run every 300 seconds.

It happends every polling interval. spine ends up on all pollers without apparent problems, but cmd.php keeps running on main. Most of the actual polling is run on one of the remote pollers, main poller have no device associated (now it has one for testing).

For a typical polling this is the final outout I see on the log file of each poller
poller 01:
2021/11/08 14:30:41 - SYSTEM STATS: Time:38.2217 Method:spine Processes:1 Threads:30 Hosts:170 HostsPerProcess:170 DataSources:20848 RRDsProcessed:0
poller 02:
2021/11/08 14:30:07 - SYSTEM STATS: Time:5.1605 Method:spine Processes:1 Threads:30 Hosts:27 HostsPerProcess:27 DataSources:468 RRDsProcessed:0
main poller:
2021/11/08 14:30:03 - SPINE: Poller[1] PID[5695] PT[140715085393664] Time: 0.5676 s, Threads: 30, Devices: 1
2021/11/08 14:30:03 - SPINE: Poller[1] PID[5731] PT[139950535016192] Time: 0.5673 s, Threads: 30, Devices: 2
after a time i'll see a timeout line, but weirdly more like 10 minutes after start (not 5, but log says timeout after 299 seconds):
2021/11/08 14:39:55 - ERROR PHP ERROR: Maximum execution time of 299 seconds exceeded in file: /opt/cacti/lib/poller.php on line: 462
2021/11/08 14:39:55 - CMDPHP PHP ERROR Backtrace: (CactiShutdownHandler())

When this apparently zombie poller.php are running mysql cpu usage goes up (and stays there as for the time a process is killed up to new 3 process will be up).

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3521 mysql 20 0 2827852 837396 24028 S 103,6 20,83 56:01.36 /usr/sbin/mysqld --defaults-file=/etc/my.cnf --user=mysql
5659 wwwrun 20 0 353616 224988 17760 R 52,32 5,596 2:54.58 php /opt/cacti/poller.php
5815 wwwrun 20 0 329912 201416 17748 S 42,72 5,009 0:23.14 php /opt/cacti/poller.php

I stopped cron calls to poller.php on all 3 servers, changed poller to cmd.php and tried to run: poller --debug --force but I see no logging lines. Tried this also with debug level set to maximum (DEVEL) on the webui settings, but made no difference.

sudo -uwwwrun php /opt/cacti/poller.php --debug --force
2021/11/08 12:05:08 - POLLER: Poller[1] PID[3135] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '72.33', Max Runtime '298', Poller Runs: '1'
2021/11/08 12:05:09 - POLLER: Poller[1] PID[3135] WARNING: Poller Output Table not Empty. Issues: 0, DS[64, 74, 23793, 24258, 864, 20776, 24410, 22623, 8301, 8302, 23817, 23817, 23818, 23818, 23819, 23819, 20776, 8301, 20775, 24411]
2021/11/08 12:05:10 - POLLER: Poller[1] PID[3135] DEBUG: About to Spawn a Remote Process [CMD: /usr/local/spine/bin/spine, ARGS: -C '/etc/cacti/spine.conf' --poller=1 --first=0 --last=0 --mibs]
Waiting on 1 of 1 pollers.
PHP Fatal error: Maximum execution time of 299 seconds exceeded in /opt/cacti/lib/database.php on line 287

I don't have a clue what is it doing until timeout. Don't see information on log or console.

I'm running SLES15SP2 (main and 1 poller) and Debian 11 (another poller).

[bmfmancini]

what is the output of strace on one of the zombie process

try running strace -p (poller.php process) and show some of the output

Also while the zombie poller.php is running please show the output of this mysql statment

select * from poller_output where output = "U";

[nicolatron]

I've attached 2 files with the requested information.
I've noted now more problems. After like 4-5 polling runs I have this errors:
2021/11/09 08:10:03 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2021/11/09 08:10:07 - SPINE: Poller[1] PID[10032] PT[140558032713472] ERROR: SQL Failed! Error:'1114', Message:'The table 'poller_output' is full', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (24075, 'ping', FROM_UNIXTIME(1636441805), '1.73'),(22963, '5min_cpu', FROM_UNIXTIME(1636441805), '6') ON DUPLICATE KEY UPDATE output=VALUES(output)'
2021/11/09 08:10:58 - ERROR PHP ERROR: Maximum execution time of 299 seconds exceeded in file: /opt/cacti/lib/poller.php on line: 462
2021/11/09 08:10:58 - CMDPHP PHP ERROR Backtrace: (CactiShutdownHandler())

MySQL is configured with no upper limit, and the filesystem in wich mysql files are creaed still has 5,6GB free space. Relevant configution:
collation_server=utf8mb4_unicode_ci
character_set_server=utf8mb4
max_connections=250
max_allowed_packet=16777216
max_heap_table_size=70M
tmp_table_size=70M
join_buffer_size=135M
innodb_file_per_table=ON
innodb_file_format=Barracuda
innodb_large_prefix=1
innodb_buffer_pool_size=1024M
innodb_doublewrite=OFF
innodb_lock_wait_timeout=50
innodb_flush_method=O_DIRECT
innodb_flush_log_at_timeout=3
innodb_read_io_threads=8
innodb_write_io_threads=4
innodb_buffer_pool_instances=9
innodb_io_capacity=200
innodb_io_capacity_max=2000
innodb_use_atomic_writes=ON
innodb_data_file_path = ibdata1:10M:autoextend
skip-name-resolve

/dev/mapper/vg_root-lv_root 12G 5,6G 5,6G 51% /

And before anyone loses more time figuring out what may be the problem, i think i should include more background information. I reported this as a 1.2.19 problem. But really problem started when I was running 1.2.18 (cacti) + 1.2.16 (spine). 2 weeks ago after I spent many hours creating a new error graph template with % of discard/errors, and finally understood (or kind of) CDEF, RPN etc. I went in a homonegnization frenzy and run a lot of both, sync graph to template, reapply suggested names and all possible cli/(poller_reindex_hosts.php, rebuild_poller_cache, repair_database, repair_templates...). Some time (1 day maybe) after that I noticed that my weathermaps always had 0 value on LINKs (weirdly (for me) cpu counters related to NODE and not LINKs were alright, TARGET of LINK seemed right and pointing to existing rrd files which existed as local_id on database), that was the moment I decided to move to 1.2.19 in the hope everything will fix itself. Now I have all plugins (only use weathermap) disabled. This database had been upgraded all the way from 0.8.something, maybe I restarted from scratch at some point in 1.something, not sure, but anyway it has seen a lot of upgrades and all kind of host and graph template imports.

Another possible meaningfull piece of data. When I reach the poller_output is full state. poller output is fixed at this value:

MariaDB [cacti]> select count() from poller_output;
+----------+
| count() |
+----------+
| 180810 |
+----------+
1 row in set (0,000 sec)

MariaDB [cacti]> show table status like 'poller_output%';
+------------------------------------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+--------------------+----------+--------------------+---------+------------------+-----------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment | Max_index_length | Temporary |
+------------------------------------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+--------------------+----------+--------------------+---------+------------------+-----------+
| poller_output | MEMORY | 10 | Fixed | 180810 | 286 | 52078400 | 51705368 | 9312191 | 0 | NULL | 2021-11-08 16:45:02 | NULL | NULL | utf8mb4_unicode_ci | NULL | row_format=DYNAMIC | | 0 | N |
| poller_output_boost | InnoDB | 10 | Dynamic | 4189381 | 76 | 318767104 | 0 | 0 | 7340032 | NULL | 2021-11-08 15:05:02 | 2021-11-09 08:25:17 | NULL | utf8mb4_unicode_ci | NULL | row_format=DYNAMIC | | 0 | N |
| poller_output_boost_local_data_ids | MEMORY | 10 | Fixed | 0 | 9 | 0 | 13762557 | 0 | 0 | NULL | 2021-11-08 16:48:22 | NULL | NULL | utf8mb4_unicode_ci | NULL | | | 0 | N |
| poller_output_boost_processes | InnoDB | 10 | Dynamic | 2 | 8192 | 16384 | 0 | 0 | 0 | 3 | 2020-02-13 13:09:52 | NULL | NULL | utf8mb4_unicode_ci | NULL | row_format=DYNAMIC | | 0 | N |
| poller_output_realtime | InnoDB | 10 | Dynamic | 0 | 0 | 16384 | 0 | 32768 | 0 | NULL | 2021-09-20 12:16:30 | NULL | NULL | utf8mb4_unicode_ci | NULL | | | 0 | N |
+------------------------------------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+--------------------+----------+--------------------+---------+------------------+-----------+
5 rows in set (0,001 sec)

MariaDB [cacti]> show create table poller_output;
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| poller_output | CREATE TABLE poller_output (
local_data_id int(10) unsigned NOT NULL DEFAULT 0,
rrd_name varchar(19) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
time timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
output varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '',
PRIMARY KEY (local_data_id,rrd_name,time) USING BTREE
) ENGINE=MEMORY DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci ROW_FORMAT=DYNAMIC |
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0,000 sec)

Thank you very much!

select_result.txt
strace_output.txt

[nicolatron]

I've raised max_heap_size to 140M did a reboot and more things are happening in the log (during boost write-to-disk process).
2021/11/09 09:11:05 - CMDPHP ERROR: A DB Row Failed!, Error: Error writing file '/tmp/MYEGVfxQ' (Errcode: 28 "No space left on device")
Asked for 30G more disk for the VM.

[nicolatron]

Moved mysql files to another partition, fixed the space problem. Also raised max_heap_table size to (first 140M which gave a max row size of 361195 rows on poller_output) 500M which should be enougth to accomodate the million row set as maximum Records in Settings->Performance. I swear max_heap_table_size was 1G some weeks ago, but oh well, I also updated the systems, maybe the mysql config file was overwritten at some point.
Also cleaned a lot of polling errors, like datasource/graphs with wrong values, or polling for non-existant OIDs.
Situation is still like explained on first post, poller keeps running on main poller until timeout.

Attrached refreshed versions of query and strace:
select_result2.txt
strace2.txt

[nicolatron]

Execution of cacti/cli/poller_output_empty.php also gets hanged. Seems to enter an infinite while loop.
Setting inside the loop a print "nico $rrdtool_pipe rrds_processed=$rrds_processed\n"; results in:
OV01LNXVZ4:/opt/cacti # cli/poller_output_empty.php
nico Resource id #43 rrds_processed=0
nico Resource id #43 rrds_processed=0
[...]
Maybe this is related with the problems i'm having, seems like the function process_poller_output in lib/poller.php is having problems. At the end nothing seems to be deleted from poller_output table.
Trying to see where it's getting stuck.

[bmfmancini]

Yea I also see the same as well when you try to run poller output empty it just hangs I'm having the exact same problem you are It seems the output table doesn't clear on my case any device that times out clogs that table and poller.php never completes

…

On Wed., Nov. 10, 2021, 07:48 Nicolás Victorero, ***@***.***> wrote: Execution of cacti/cli/poller_output_empty.php also gets hanged. Seems to enter an infinite while loop. Setting inside the loop a print "nico $rrdtool_pipe rrds_processed=$rrds_processed\n"; results in: OV01LNXVZ4:/opt/cacti # cli/poller_output_empty.php nico Resource id #43 <#43> rrds_processed=0 nico Resource id #43 <#43> rrds_processed=0 [...] Maybe this is related with the problems i'm having, seems like the function process_poller_output in lib/poller.php is having problems. At the end nothing seems to be deleted from poller_output table. Trying to see where it's getting stuck. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4460 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGEXTEKKKSE6AD6UNJZAZDULJS3HANCNFSM5HSWIDQQ> .

[nicolatron]

Seems like this conditions are very rarely met:
if (isset($rrd_update_array[$rrd_path]['times'][$unix_time])) {
if ($item['rrd_num'] <= cacti_sizeof($rrd_update_array[$rrd_path]['times'][$unix_time])) {

The rrd_update_array contains:

(
    [/opt/cacti/rra/core-vss_cisco_memfree_32.rrd] => Array
        (
            [local_data_id] => 32
            [times] => Array
                (
                    [1636550705] => Array
                        (
                            [cisco_memfree] => 755039240
                        )

                )

        )

And the $items for that rrd file path:
nico end rrd_update_array

Array
(
    [output] => 755039240
    [time] => 2021-11-10 14:25:05
    [unix_time] => 1636550705
    [local_data_id] => 32
    [data_template_id] => 53
    [rrd_path] => /opt/cacti/rra/core-vss_cisco_memfree_32.rrd
    [rrd_name] => cisco_memfree
    [rrd_num] => 1
)

and for that r un rrd_update_array size was 1266 so I think both conditions were met. But i've put another comment inside the loop, where the DELETE sentence should be generated, and it's but, k neet to be 10000 also...mmm

Still trying to figure things out. output for a 3 seconds log from poller_output.php redirected to a file print_r ing $rrd_update_array and $item is 55MB long.

[bmfmancini]

By chance are you seeing these messeges in your OS logs

[1124946.932296] php[4171170]: segfault at 7fff25bd3fe0 ip 00007fa5fe087d64 sp 00007fff25bd3fd0 error 6 in libpcre2-8.so.0.7.1[7fa5fe043000+83000]
[1124946.935261] Code: 10 00 00 48 83 0c 24 00 4c 39 dc 75 ef 48 81 ec 88 02 00 00 4c 8b a4 24 c0 52 00 00 48 89 4c 24 38 48 89 c8 49 89 ff 49 89 f6 <4c> 89 4c 24 10 48 01 f0 49 89 d2 64 48 8b 1c 25 28 00 00 00 48 89
[1125310.011036] php[4174275]: segfault at 7ffea47a7ff8 ip 00007fbb3572dd56 sp 00007ffea47a7fc0 error 6 in libpcre2-8.so.0.7.1[7fbb356e9000+83000]
[1125310.013465] Code: 55 53 4c 8d 9c 24 00 b0 ff ff 48 81 ec 00 10 00 00 48 83 0c 24 00 4c 39 dc 75 ef 48 81 ec 88 02 00 00 4c 8b a4 24 c0 52 00 00 <48> 89 4c 24 38 48 89 c8 49 89 ff 49 89 f6 4c 89 4c 24 10 48 01 f0

I also see when I run poller.php --force -d it eventually gets to a seg fault and the php script server crashes

try turning off cron/cactid and run the poller manually the same see if you get the same thing

[bmfmancini]

Oh BTW which OS and php version are you running ?

I tried using the previous version of poller. PHP and same issue is happening

I'm seeing segfaults for pcre2 on rhel 8 when poller.php exits

[nicolatron]

SUSE Linux Enterprise Server 15 SP2
RRDtool 1.7.0
Ver 15.1 Distrib 10.4.17-MariaDB
PHP 7.4.6 (cli) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies

Running sudo -uwwwrun php -q /opt/cacti/cli/poller.php --force --debug ends after 299 seconds

PHP Fatal error: Maximum execution time of 299 seconds exceeded in /opt/cacti/lib/poller.php on line 540

I see nothing like your errors in /var/log/messages dmesg output or /opt/cacti/log/cacti.log
I've decided to start my cacti database from scratch, thinking if there is a easy way to recover or export/import at least devices with snmp configurations and user authorization (authentication is done against an external LDAP).
In my system it's not only that poller process are not deleting poller_output table, the values I'm getting from many
graphs seems to be broken, like registering the same information from in and out counters.
Something went wrong, at some point of my many changes of the last weeks, os upgrades, and graph, device and template cleanup/reapply etc. And I'm not sure when or why things broke.
Will have a lot of work to do remapping graph ids to weathermap configs, not to say the historical data, but oh well...
I'll wait a few days and close this issue, I really don't think it is really 1.2.19 related anyway.

[bmfmancini]

Hold off on closing for now I am also seeing the same thing you are I'm on rhel not suse but I am running php 7.4

…

On Fri., Nov. 12, 2021, 02:30 Nicolás Victorero, ***@***.***> wrote: SUSE Linux Enterprise Server 15 SP2 RRDtool 1.7.0 Ver 15.1 Distrib 10.4.17-MariaDB PHP 7.4.6 (cli) ( NTS ) Copyright (c) The PHP Group Zend Engine v3.4.0, Copyright (c) Zend Technologies Running sudo -uwwwrun php -q /opt/cacti/cli/poller.php --force --debug ends after 299 seconds PHP Fatal error: Maximum execution time of 299 seconds exceeded in /opt/cacti/lib/poller.php on line 540 I see nothing like your error in /var/log/messages dmesg output or /opt/cacti/log/cacti.log I've decided to start my cacti database from scratch, thinking if there is a easy way to recover or export/import at least devices with snmp configurations and user authorization (authentication is done against an external LDAP). In my system it's not only that poller process are not deleting poller_output table, the values I'm getting from many graphs seems to be broken, like registering the same information from in and out counters. Something went wrong, at some point of my many changes of the last weeks, os upgrades, and graph, device and template cleanup/reapply etc. And I'm not sure when or why things broke. Will have a lot of work to do remapping graph ids to weathermap configs, not to say the historical data, but oh well... I'll wait a few days and close this issue, I really don't think it is really 1.2.19 related anyway. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4460 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGEXTHRXV5EFI4X5S4NAWLULS7AFANCNFSM5HSWIDQQ> .

[bmfmancini]

Looking at the attachment you out of the select statement from poller_output you have the same thing in seeing entries from 3 polling cycles and only the U values are sticking around

[bmfmancini]

ok this is what I did and it seems to be working very good

I used the lib/poller.php from 1.2.16 ( I had it downloaded already I am sure 1.2.18 will work too)
polling is working just fine now