Continued timeout of registered processes #4271
Comments
|
Timeout is 300 seconds, yet this system with a 30 second collect frequency continually generate this error.
|
|
I am seeing the same issue even on 1.2.17
…On Tue., May 18, 2021, 20:29 TheWitness, ***@***.***> wrote:
Timeout is 300 seconds, yet this system with a 30 second collect frequency
continually generate this error.
[image: image]
<https://user-images.githubusercontent.com/1439914/118739509-a6f89680-b817-11eb-8d3d-d3749011b0a9.png>
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#4271 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADGEXTEO3T6HVPBD3OR2XVDTOMA5PANCNFSM45DRQE4A>
.
|
|
Going to move the timeout entirely into SQL as there seems to be continual problems with PHP and the web server. |
TheWitness
added a commit
that referenced
this issue
May 19, 2021
By moving the calculation fully under SQL, this should solve the problem.
|
Please test the commit that just went in. |
|
Ok will test it in the morning
…On Tue., May 18, 2021, 20:38 TheWitness, ***@***.***> wrote:
Please test the commit that just went in.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#4271 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADGEXTFYTVQWTEISNDJ5NMLTOMB7RANCNFSM45DRQE4A>
.
|
|
OK put it in our lab will monitor and let you know |
|
@TheWitness still seeing the same behaviour Made sure I grabbed the latest commits diff poller.php poller.php.0519
1931c1931
< $r = db_fetch_row_prepared('SELECT *, IF(UNIX_TIMESTAMP(started) + timeout < UNIX_TIMESTAMP(), 1, 0) AS timeout_exceeded
---
> $r = db_fetch_row_prepared('SELECT *
1942c1942
< } elseif ($r['timeout_exceeded']) {
---
> } elseif (strtotime($r['started']) + $r['timeout'] < time()) { |
|
I've made a change to be a bit more verbose on the debug, plus to ensure we definitely insert a start time and do not rely on MySQL to set it. |
|
Cool I will test it in the am |
|
OK so testing this out the extra verbose code really helps narrow this down |
|
I think the error here is that CURRENT_TIMESTAMP is a reserved word probably and isn't escaped. |
|
Try that commit and see if it resolves things for you |
|
All good now |
|
Are you still seeing events being timeout ? |
|
I will check on Tuesday but so far so good
…On Sun., May 23, 2021, 15:13 Mark Brugnoli-Vinten, ***@***.***> wrote:
Are you still seeing events being timeout ?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#4271 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADGEXTGLUWAMNXSW4HZK2XLTPFHUXANCNFSM45DRQE4A>
.
|
|
A bit late with the update but yea not a single timeout message since the update ! |
|
This issue was not actually resolved. I've made an update this morning. |
TheWitness
added a commit
that referenced
this issue
Aug 28, 2021
The logic that was placed into the service check was flawed resulting in incorrect timestamps in the database. I've yet to check the timeout handling.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Describe the bug
For some unknown reason, processes are still continuing to be killed by the register_process function though they have not timed out.
To Reproduce
Reboot a large number of servers, switches and routers enough where the re-index will take several minutes for all devices.
When the next poller run starts not the re-index log entries as in the image above
Note at that at the next poller start, you see the poller command script being restarted.
Expected behavior
The poller should respect the timeout.
The text was updated successfully, but these errors were encountered: