Poller recovery starts multiple processes and fails to recover properly

[bernisys]

Describe the bug

First of all, this probably appears only in really large setups, where the recovery is taking more time than a polling cycle.

When connectivity returns after an outage in a remote poller setup, the poller recovery does not work properly.
It spawns multiple processes and causes a lot of DB stress, because it does not take into account that a process is already running.

I also found the actual flaw in the code and i think i was able to fix it.

To Reproduce

Let a really large (in terms of data sources) remote poller go into "Heartbeat" mode and let it recover again.
You will see multiple processes running after a while, which are trying all the same: starting the data sync from scratch over and over again.

Expected behavior

Poller recovery should run only once.
Every subsequent call should detect that a process is running already.

Additional context

The problem is, that the "recovery_pid" is nowhere inserted into the corresponding table.
It is only deleted at one point in the code.
This way the subsequent processes always see an empty value (because there is no "recovery_pid" entry in the "settings" table) and will start the process all over again.

[bmfmancini]

Hey I have found the same thing in 1.2.16 if the poller is off for a longer period of time it will just keep in recovery state in my case the test was showing 15M records to be recovered but poller recovery spawned multiple processes

…

On Thu., Apr. 8, 2021, 08:10 BerndP, ***@***.***> wrote: Describe the bug First of all, this probably appears only in really large setups, where the recovery is taking more time than a polling cycle. When connectivity returns after an outage in a remote poller setup, the poller recovery does not work properly. It spawns multiple processes and causes a lot of DB stress, because it does not take into account that a process is already running. I also found the actual flaw in the code and i think i was able to fix it. To Reproduce Let a really large (in terms of data sources) remote poller go into "Heartbeat" mode and let it recover again. You will see multiple processes running after a while, which are trying all the same: starting the data sync from scratch over and over again. Expected behavior Poller recovery should run only once. Every subsequent call should detect that a process is running already. Additional context The problem is, that the "recovery_pid" is nowhere inserted into the corresponding table. It is only deleted at one point in the code. This way the subsequent processes always see an empty value (because there is no "recovery_pid" entry in the "settings" table) and will start the process all over again. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4215>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGEXTE4NGMN5JTWE4APF4DTHWMMBANCNFSM42SYI45A> .

[TheWitness]

You need to move that pull request to the 1.2.x branch. I've closed it for now.

[bernisys]

Sure, will do so, sorry didn't know.

[bernisys]

That should be better now.

@bmfmancini - could you please verify if that works also for you?

In our case i have seen lately that the PIDs that are written into the table differ from the ones of the runing processes.
But still, now there is just one process running in our setup.

[bmfmancini]

Sure will do once the pr is committed I'll pull it down

…

On Thu., Apr. 8, 2021, 09:33 BerndP, ***@***.***> wrote: That should be better now. @bmfmancini <https://github.com/bmfmancini> - could you please verify if that works also for you? In our case i have seen lately that the PIDs that are written into the table differ from the ones of the runing processes. But still, now there is just one process running in our setup. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4215 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGEXTDNZTRMQBDDCUGOWCLTHWWB5ANCNFSM42SYI45A> .

[bmfmancini]

Initial tests are looking good

[bernisys]

Somehow i think this is not yet the complete truth .. I have observed the recovery process for a while now, it was still not completely working as it should be.

The INSERT statement that i added is not always working properly, specially when the process times out and is killed. (Which should not happen IMHO - the script should run, otherwise the initial master->poller replication of those 4 tables might be happening over and over again, which costs valuable time and lets the output table fill.)

I have now added "ON DUPLICATE KEY UPDATE value=VALUES(value)" into the patch - it's looking way better now.
Replication starts with pulling these 4 tables to the poller, then it does something in the background silently ...
I would wish for more verbosity here by default, for example seeing lines like this every once in a while would be nice:
"pushing out of <max_n> values back to main"
Then we can see much better how the sync is proceeding, and this should be not just debug output but default output, as it occurs not in a debug session but after an emergency situation.

Also i am still a bit worried about this initial "undefined offset" at the start, this should be caught and logged properly, so that we know what this kind of error really means. In fact there are a lot of different other errors that should be explicitly handled. You might know the answer, but someone new to the tool is certainly lost :)

$ php ./poller_recovery.php
PHP Notice:  Undefined offset: 1 in /opt/SP/cacti/cacti/lib/database.php on line 415
2021-04-08 19:27:48 - WEBUI NOTE: Table data_local Replicated to Poller With 68208 Rows Updated
2021-04-08 19:28:07 - WEBUI NOTE: Table graph_local Replicated to Poller With 66929 Rows Updated
2021-04-08 19:29:45 - WEBUI NOTE: Table host_snmp_cache Replicated to Poller With 247247 Rows Updated
2021-04-08 19:29:45 - WEBUI NOTE: Table poller_reindex Replicated to Poller With 491 Rows Updated
PHP Fatal error:  Maximum execution time of 300 seconds exceeded in /opt/SP/cacti/cacti/poller_recovery.php on line 217

But the PID of the then new spawned process is now correct in the DB.
Will prepare that fix now and offer another pull.

Have to get a bit more used to the internals.