Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Undefined index: timeout_exceed /lib/poller.php on line: 1953 #4440

Closed
jdcoats opened this issue Oct 22, 2021 · 38 comments
Closed

Undefined index: timeout_exceed /lib/poller.php on line: 1953 #4440

jdcoats opened this issue Oct 22, 2021 · 38 comments
Labels
bug Undesired behaviour resolved A fixed issue
Milestone

Comments

@jdcoats
Copy link

jdcoats commented Oct 22, 2021

2021/10/22 10:35:18 - POLLER: Poller[Main Poller] PID[8312] ERROR: Process being killed due to timeout! (commands, master, 1, Process 23662, Time , Timeout 300, Timestamp 1634913318)
2021/10/22 10:35:18 - CMDPHP PHP ERROR NOTICE Backtrace: (/poller_commands.php[121]:register_process_start(), /lib/poller.php[1953]:CactiErrorHandler())
2021/10/22 10:35:18 - ERROR PHP NOTICE: Undefined index: timeout_exceed in file: /var/www/localhost/htdocs/cacti/lib/poller.php on line: 1953
@jdcoats jdcoats added bug Undesired behaviour unverified Some days we don't have a clue labels Oct 22, 2021
@jdcoats
Copy link
Author

jdcoats commented Oct 22, 2021

I'm also stuck with constant recaching of 100+ devices.

NOTE: Recache Event Detected for Device
RECACHE ASSERT FAILED:
 

@TheWitness TheWitness added this to the v1.2.19 milestone Oct 22, 2021
@TheWitness TheWitness removed the unverified Some days we don't have a clue label Oct 22, 2021
TheWitness added a commit that referenced this issue Oct 22, 2021
Undefined index: timeout_exceed /lib/poller.php on line: 1953
@TheWitness TheWitness added the resolved A fixed issue label Oct 22, 2021
@TheWitness
Copy link
Member

Please test and close if resolved. Thanks!

@jdcoats
Copy link
Author

jdcoats commented Oct 22, 2021

I will as soon as I can recover whatever is going on. I updated Cacti & Spine this morning and was getting recache events constantly for many devices. I ended up rolling back to yesterdays Cacti and Spine from Oct 1. I'm missing many datasources now. Still trying to figure out how to calm the storm.

2021/10/22 11:52:01 - POLLER: Poller[Main Poller] PID[22424] WARNING: Poller Output Table not Empty. Issues: 98, DS[Missing Datasource 71240, Missing Datasource 71245, Missing Datasource 71225, Missing Datasource 71152, Missing Datasource 71310, Missing Datasource 71285, Missing Datasource 71275, Missing Datasource 71260, Missing Datasource 71305, Missing Datasource 71340, Missing Datasource 71290, Missing Datasource 71132, Missing Datasource 71325, Missing Datasource 71196, Missing Datasource 71142, Missing Datasource 71112, Missing Datasource 71235, Missing Datasource 71270], Additional Issues Remain. Only showing first 20

@jdcoats
Copy link
Author

jdcoats commented Oct 22, 2021

for example 100 PDU's without snmp interface stats on the device or graphs started doing this:

2021/10/22 11:58:01 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-4052-PDU-02] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
--
2021/10/22 11:57:59 - PCOMMAND Device[C-Mustian-4052-PDU-02] NOTE: Recache Event Detected for Device
2021/10/22 11:57:59 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-4052-PDU-01] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:57 - PCOMMAND Device[C-Mustian-4052-PDU-01] NOTE: Recache Event Detected for Device
2021/10/22 11:57:57 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-4033-PDU-03] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:56 - PCOMMAND Device[C-Mustian-4033-PDU-03] NOTE: Recache Event Detected for Device
2021/10/22 11:57:56 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-4033-PDU-02] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:54 - PCOMMAND Device[C-Mustian-4033-PDU-02] NOTE: Recache Event Detected for Device
2021/10/22 11:57:53 - PCOMMAND Device[C-Mustian-4033-PDU-01] NOTE: Recache Event Detected for Device
2021/10/22 11:57:53 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-3052-PDU-03] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:51 - PCOMMAND Device[C-Mustian-3052-PDU-03] NOTE: Recache Event Detected for Device
2021/10/22 11:57:51 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-3052-PDU-02] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:49 - PCOMMAND Device[C-Mustian-3052-PDU-02] NOTE: Recache Event Detected for Device
2021/10/22 11:57:49 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-3052-PDU-01] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:47 - PCOMMAND Device[C-Mustian-3052-PDU-01] NOTE: Recache Event Detected for Device
2021/10/22 11:57:47 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-3033-PDU-03] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:45 - PCOMMAND Device[C-Mustian-3033-PDU-03] NOTE: Recache Event Detected for Device
2021/10/22 11:57:45 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-3033-PDU-02] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:43 - PCOMMAND Device[C-Mustian-3033-PDU-02] NOTE: Recache Event Detected for Device
2021/10/22 11:57:42 - PCOMMAND Device[C-1POB-4-SW48-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 11:57:42 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-3033-PDU-01] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:41 - PCOMMAND Device[C-Mustian-3033-PDU-01] NOTE: Recache Event Detected for Device
2021/10/22 11:57:41 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-1470-PDU-03] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:40 - PCOMMAND Device[C-Mustian-1470-PDU-03] NOTE: Recache Event Detected for Device
2021/10/22 11:57:40 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-1470-PDU-02] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:39 - PCOMMAND Device[C-Mustian-1470-PDU-02] NOTE: Recache Event Detected for Device
2021/10/22 11:57:39 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-1470-PDU-01] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:38 - PCOMMAND Device[C-Mustian-1470-PDU-01] NOTE: Recache Event Detected for Device
2021/10/22 11:57:38 - REINDEX WARNING: Sort Field has Changed for Device[C-Mustian-1459-PDU-03] and DQ[SNMP - Interface Statistics]. Old Sort:ifName, New Sort:ifDescr. Re-mapping issues may occur!
2021/10/22 11:57:36 - PCOMMAND Device[C-Mustian-1459-PDU-03] NOTE: Recache Event Detected for Device

@TheWitness
Copy link
Member

Well, there was a broken spine a few days ago around constant recaches. It was fixed a day or so ago. So, if you update spine, make sure you get the latest. Also the latest spine depends on the column snmp_sysUpTimeInstance in the host table being bigint unsigned. You might have to manually update that column.

@TheWitness
Copy link
Member

The flipping of ifName to ifDescr is almost like you have modified the resource XML file to use ifDescr as the sort key. I hope that was not a part of the package. Checking from my build machine.

@TheWitness
Copy link
Member

The packaged version of interface.xml is correct.

@jdcoats
Copy link
Author

jdcoats commented Oct 22, 2021

using spine built this morning. using the default packages, even reinstalled all of them. I did modify int to bigint

ALTER TABLE host MODIFY COLUMN snmp_sysUpTimeInstance bigint(20) unsigned not null default "0";
ALTER TABLE automation_devices MODIFY COLUMN sysUpTime bigint(20) unsigned not null default "0";

problem persists

@jdcoats
Copy link
Author

jdcoats commented Oct 22, 2021

also still getting that process killed due to timeout but without the undefined index :)

2021/10/22 17:16:19 - PCOMMAND Device[M-Bradfordville-SW48-01] NOTE: Recache Event Detected for Device
2021/10/22 17:16:17 - PCOMMAND Device[M-Blountstown-Rehab-SW24-01] NOTE: Recache Event Detected for Device
2021/10/22 17:16:17 - POLLER: Poller[Main Poller] PID[5170] ERROR: Process being killed due to timeout! (commands, master, 1, Process 20129, Time 1634937020, Timeout 300, Timestamp 1634937377)
2021/10/22 17:16:17 - SYSTEM STATS: Time:16.3599 Method:spine Processes:2 Threads:40 Hosts:1774 HostsPerProcess:887 DataSources:121798 RRDsProcessed:0
2021/10/22 17:16:14 - PCOMMAND Device[H-1N-36-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:16:11 - PCOMMAND Device[H-1N-33-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:16:07 - PCOMMAND Device[H-GS-152-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:16:05 - PCOMMAND Device[H-CompRoom-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:16:02 - PCOMMAND Device[H-GN-50-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:16:00 - PCOMMAND Device[H-Facilities-STCK-01] NOTE: Recache Event Detected for Device
2021/10/22 17:15:59 - PCOMMAND Device[M-Carrabelle-SW24-01] NOTE: Recache Event Detected for Device
2021/10/22 17:15:58 - PCOMMAND Device[M-Apalachicola-SW24-01] NOTE: Recache Event Detected for Device
2021/10/22 17:15:55 - PCOMMAND Device[H-GB-41-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:15:51 - PCOMMAND Device[H-GA-38-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:15:49 - PCOMMAND Device[H-GA-2-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:15:47 - PCOMMAND Device[H-MDF-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:15:45 - PCOMMAND Device[H-BN-65-STCK-01.TMH.ORG] NOTE: Recache Event Detected for Device
2021/10/22 17:15:44 - PCOMMAND Device[M-FSU-SW24-01] NOTE: Recache Event Detected for Device
2021/10/22 17:15:41 - PCOMMAND Device[M-Heart-Rhythm-STCK-01] NOTE: Recache Event Detected for Device

@jdcoats
Copy link
Author

jdcoats commented Oct 22, 2021

poller_commands hanging up for 5+ minutes

root     19635  1.9  0.1 125416 86792 ?        S    17:21   0:03 /usr/bin/php -q /var/www/localhost/htdocs/cacti/poller_commands.php --poller=1

I deleted all of the PDU's mentioned above and re-imported them and that issue went away. The constant Recache Events are still happening.

@jdcoats
Copy link
Author

jdcoats commented Oct 22, 2021

I wish I could help more but I'd say the caching is definitely more persistent now :)

image

@jdcoats
Copy link
Author

jdcoats commented Oct 22, 2021

Boost is far faster now though

image
image
image

@bmfmancini
Copy link
Member

bmfmancini commented Oct 22, 2021 via email

@jdcoats
Copy link
Author

jdcoats commented Oct 22, 2021

i thought i was on to the fact that it was udp ping but then I found icmp ping on a few. I "think" all are ping & snmp though

@bmfmancini
Copy link
Member

bmfmancini commented Oct 22, 2021 via email

@jdcoats
Copy link
Author

jdcoats commented Oct 23, 2021

I just rebooted 3 switch stacks and didn't get notified for all devices that should and do always notify otherwise. Things are getting skipped.

@TheWitness
Copy link
Member

The math appears wrong. There may be a problem with some of the data being stored as long vs. normal integers in spine. Though I have not seen it, it could simply be due to the amount of uptime. I'll do another code review today, but I have a family gathering this afternoon, and shopping to do now. So, my efforts may not happen till Sunday.

@TheWitness
Copy link
Member

@jdcoats are you also running the full 1.2.x branch of Cacti?

@jdcoats
Copy link
Author

jdcoats commented Oct 23, 2021 via email

@TheWitness
Copy link
Member

Okay, I just did a push to poller.c. Test it out and let me know.

@jdcoats
Copy link
Author

jdcoats commented Oct 23, 2021

It appears to still be the same issue. I know you said you were busy today, I can make myself available for a bit tomorrow if you need to see it. Thanks!

@TheWitness
Copy link
Member

Can you please upload the resource XML. It seems the assert value is the "traditional" sysUpTimeIndex and the sysUpTimeIndex is using the new value.

@TheWitness
Copy link
Member

Okay, the issue was resolved in lib/poller.php in the Cacti 1.2.x branch. Update that file and report back. May take two poller cycles to clear.

@TheWitness
Copy link
Member

If you just updated, update again. Forgot to multiply by 100.

@jdcoats
Copy link
Author

jdcoats commented Oct 23, 2021

still appears to be the same, I waited 10 polling cycles.

@TheWitness
Copy link
Member

Upload that XML.

@jdcoats
Copy link
Author

jdcoats commented Oct 23, 2021 via email

@TheWitness
Copy link
Member

resource/snmp_queries

@jdcoats
Copy link
Author

jdcoats commented Oct 23, 2021

didn't know which one you wanted so you got them all :)
snmp_queries.zip

@jdcoats
Copy link
Author

jdcoats commented Oct 23, 2021

the interface.xml in the attached was from the package re-install but I have tried both the default one from 1.2.x branch and the packaged one from the branch, they do differ slightly.

TheWitness added a commit that referenced this issue Oct 24, 2021
The wrong OID uptime is inserted into the poller_reindex table.
@TheWitness
Copy link
Member

So, update lib/poller.php again, and then truncate the poller_reindex table. It'll cause all devices to be re-indexed, so prepare for a surge of re-index requests. You can control the number of re-indexes by just deleting the values for the host_id in question. After which, track the assert_value in that table. It should 'always' have two trailing zero's. Provide some feedback.

@TheWitness
Copy link
Member

You should see something like the following image. All the arg1 should be the sysUptimeInstance OID and the last two digits should all be zero unless the devices does not support the alternate sysUptimeInstance as a seconds integer.

image

@jdcoats
Copy link
Author

jdcoats commented Oct 24, 2021

MariaDB [cacti]> select * from poller_reindex;
+---------+---------------+--------+---------+----+--------------+-------------------------+
| host_id | data_query_id | action | present | op | assert_value | arg1                    |
+---------+---------------+--------+---------+----+--------------+-------------------------+
|     154 |             1 |      0 |       1 | <  | 2668326700   | .1.3.6.1.2.1.1.3.0      |
|     172 |             1 |      0 |       1 | <  | 1512479600   | .1.3.6.1.2.1.1.3.0      |
|     174 |             1 |      0 |       1 | <  | 148650200    | .1.3.6.1.2.1.1.3.0      |
|     178 |             1 |      0 |       1 | <  | 1905191100   | .1.3.6.1.2.1.1.3.0      |
|     181 |             1 |      0 |       1 | <  | 657298600    | .1.3.6.1.2.1.1.3.0      |
|     181 |             4 |      0 |       1 | <  | 657298600    | .1.3.6.1.2.1.1.3.0      |
|     181 |             5 |      0 |       1 | <  | 657298700    | .1.3.6.1.2.1.1.3.0      |
|     191 |             1 |      0 |       1 | <  | 1053386800   | .1.3.6.1.2.1.1.3.0      |
|     196 |             1 |      0 |       1 | <  | 162427200    | .1.3.6.1.2.1.1.3.0      |
|     197 |             1 |      0 |       1 | <  | 163889900    | .1.3.6.1.2.1.1.3.0      |
|     208 |             1 |      0 |       1 | <  | 3485815500   | .1.3.6.1.2.1.1.3.0      |
|     210 |             1 |      0 |       1 | <  | 374321800    | .1.3.6.1.2.1.1.3.0      |
|     223 |            12 |      0 |       1 | <  | 13914395     | .1.3.6.1.6.3.10.2.1.3.0 |
|     223 |            13 |      0 |       1 | <  | 13914395     | .1.3.6.1.6.3.10.2.1.3.0 |
|     224 |             1 |      0 |       1 | <  | 13909670     | .1.3.6.1.6.3.10.2.1.3.0 |
|     224 |            12 |      0 |       1 | <  | 13909670     | .1.3.6.1.6.3.10.2.1.3.0 |
|     224 |            13 |      0 |       1 | <  | 13909670     | .1.3.6.1.6.3.10.2.1.3.0 |
|     225 |             1 |      0 |       1 | <  | 13912508     | .1.3.6.1.6.3.10.2.1.3.0 |
|     225 |            12 |      0 |       1 | <  | 13912508     | .1.3.6.1.6.3.10.2.1.3.0 |
|     225 |            13 |      0 |       1 | <  | 13912508     | .1.3.6.1.6.3.10.2.1.3.0 |
|     226 |             1 |      0 |       1 | <  | 13908743     | .1.3.6.1.6.3.10.2.1.3.0 |
|     226 |            12 |      0 |       1 | <  | 13908743     | .1.3.6.1.6.3.10.2.1.3.0 |
|     226 |            13 |      0 |       1 | <  | 13908743     | .1.3.6.1.6.3.10.2.1.3.0 |
|     227 |             1 |      0 |       1 | <  | 14315726     | .1.3.6.1.6.3.10.2.1.3.0 |
|     227 |            12 |      0 |       1 | <  | 14315726     | .1.3.6.1.6.3.10.2.1.3.0 |
|     227 |            13 |      0 |       1 | <  | 14315726     | .1.3.6.1.6.3.10.2.1.3.0 |
|     228 |             1 |      0 |       1 | <  | 14313016     | .1.3.6.1.6.3.10.2.1.3.0 |
|     228 |            12 |      0 |       1 | <  | 14313016     | .1.3.6.1.6.3.10.2.1.3.0 |
|     228 |            13 |      0 |       1 | <  | 14313016     | .1.3.6.1.6.3.10.2.1.3.0 |
|     229 |             1 |      0 |       1 | <  | 14316024     | .1.3.6.1.6.3.10.2.1.3.0 |
|     229 |            12 |      0 |       1 | <  | 14316024     | .1.3.6.1.6.3.10.2.1.3.0 |
|     229 |            13 |      0 |       1 | <  | 14316024     | .1.3.6.1.6.3.10.2.1.3.0 |
|     230 |             1 |      0 |       1 | <  | 14312992     | .1.3.6.1.6.3.10.2.1.3.0 |
|     230 |            12 |      0 |       1 | <  | 14312992     | .1.3.6.1.6.3.10.2.1.3.0 |
|     230 |            13 |      0 |       1 | <  | 14312992     | .1.3.6.1.6.3.10.2.1.3.0 |
|     231 |             1 |      0 |       1 | <  | 15518945     | .1.3.6.1.6.3.10.2.1.3.0 |
|     231 |            12 |      0 |       1 | <  | 15518945     | .1.3.6.1.6.3.10.2.1.3.0 |
|     231 |            13 |      0 |       1 | <  | 15518945     | .1.3.6.1.6.3.10.2.1.3.0 |
|     232 |             1 |      0 |       1 | <  | 15521006     | .1.3.6.1.6.3.10.2.1.3.0 |
|     232 |            12 |      0 |       1 | <  | 15521006     | .1.3.6.1.6.3.10.2.1.3.0 |
|     232 |            13 |      0 |       1 | <  | 15521006     | .1.3.6.1.6.3.10.2.1.3.0 |
|     233 |             1 |      0 |       1 | <  | 2347410      | .1.3.6.1.6.3.10.2.1.3.0 |
|     234 |            12 |      0 |       1 | <  | 90840        | .1.3.6.1.6.3.10.2.1.3.0 |
|     234 |            13 |      0 |       1 | <  | 90840        | .1.3.6.1.6.3.10.2.1.3.0 |
|     238 |             1 |      0 |       1 | <  | 26668071     | .1.3.6.1.6.3.10.2.1.3.0 |
|     241 |             1 |      0 |       1 | <  | 1624229      | .1.3.6.1.6.3.10.2.1.3.0 |
|     242 |             1 |      0 |       1 | <  | 3739787      | .1.3.6.1.6.3.10.2.1.3.0 |
|     245 |             1 |      0 |       1 | <  | 1624299      | .1.3.6.1.6.3.10.2.1.3.0 |
|     250 |             1 |      0 |       1 | <  | 6533744      | .1.3.6.1.6.3.10.2.1.3.0 |
|     261 |             1 |      0 |       1 | <  | 12253414     | .1.3.6.1.6.3.10.2.1.3.0 |
|     261 |            12 |      0 |       1 | <  | 12253414     | .1.3.6.1.6.3.10.2.1.3.0 |
|     273 |             1 |      0 |       1 | <  | 1113540      | .1.3.6.1.6.3.10.2.1.3.0 |
|     751 |             1 |      0 |       1 | <  | 18775807     | .1.3.6.1.6.3.10.2.1.3.0 |
|     751 |             4 |      0 |       1 | <  | 18775807     | .1.3.6.1.6.3.10.2.1.3.0 |
|     751 |             5 |      0 |       1 | <  | 18775807     | .1.3.6.1.6.3.10.2.1.3.0 |
|     770 |             1 |      0 |       1 | <  | 5840042      | .1.3.6.1.6.3.10.2.1.3.0 |
|     770 |             4 |      0 |       1 | <  | 5840042      | .1.3.6.1.6.3.10.2.1.3.0 |
|     770 |             5 |      0 |       1 | <  | 5840042      | .1.3.6.1.6.3.10.2.1.3.0 |
|     777 |             1 |      0 |       1 | <  | 3203781      | .1.3.6.1.6.3.10.2.1.3.0 |
|     778 |             1 |      0 |       1 | <  | 3916723      | .1.3.6.1.6.3.10.2.1.3.0 |
|     797 |             1 |      0 |       1 | <  | 1073349      | .1.3.6.1.6.3.10.2.1.3.0 |
|    1793 |             1 |      0 |       1 | <  | 146896500    | .1.3.6.1.2.1.1.3.0      |
|    1793 |             4 |      0 |       1 | <  | 146896500    | .1.3.6.1.2.1.1.3.0      |
|    1793 |             5 |      0 |       1 | <  | 146896600    | .1.3.6.1.2.1.1.3.0      |
|    1806 |             1 |      0 |       1 | <  | 1981774600   | .1.3.6.1.2.1.1.3.0      |
|    1806 |             4 |      0 |       1 | <  | 1981774600   | .1.3.6.1.2.1.1.3.0      |
|    1806 |             5 |      0 |       1 | <  | 1981774700   | .1.3.6.1.2.1.1.3.0      |
|    1807 |             1 |      0 |       1 | <  | 1839403200   | .1.3.6.1.2.1.1.3.0      |
|    1807 |             4 |      0 |       1 | <  | 1839403300   | .1.3.6.1.2.1.1.3.0      |
|    1807 |             5 |      0 |       1 | <  | 1839403300   | .1.3.6.1.2.1.1.3.0      |
|    1813 |             1 |      0 |       1 | <  | 91932200     | .1.3.6.1.2.1.1.3.0      |
|    2037 |             1 |      0 |       1 | <  | 3518175500   | .1.3.6.1.2.1.1.3.0      |
|    2593 |             1 |      0 |       1 | <  | 34915639     | .1.3.6.1.6.3.10.2.1.3.0 |
|    2596 |             1 |      0 |       1 | <  | 138798       | .1.3.6.1.6.3.10.2.1.3.0 |
|    2596 |            12 |      0 |       1 | <  | 138798       | .1.3.6.1.6.3.10.2.1.3.0 |
|    2596 |            13 |      0 |       1 | <  | 138798       | .1.3.6.1.6.3.10.2.1.3.0 |
|    2753 |             1 |      0 |       1 | <  | 35181657     | .1.3.6.1.6.3.10.2.1.3.0 |
|    2783 |             1 |      0 |       1 | <  | 12251496     | .1.3.6.1.6.3.10.2.1.3.0 |
|    2783 |            12 |      0 |       1 | <  | 12251496     | .1.3.6.1.6.3.10.2.1.3.0 |
|    2784 |             1 |      0 |       1 | <  | 12251485     | .1.3.6.1.6.3.10.2.1.3.0 |
|    2784 |            12 |      0 |       1 | <  | 12251485     | .1.3.6.1.6.3.10.2.1.3.0 |
|    2817 |             1 |      0 |       1 | <  | 1975965      | .1.3.6.1.6.3.10.2.1.3.0 |
|    2817 |            12 |      0 |       1 | <  | 1975965      | .1.3.6.1.6.3.10.2.1.3.0 |
|    2817 |            13 |      0 |       1 | <  | 1975965      | .1.3.6.1.6.3.10.2.1.3.0 |
+---------+---------------+--------+---------+----+--------------+-------------------------+
84 rows in set (0.000 sec)

@jdcoats
Copy link
Author

jdcoats commented Oct 24, 2021

MariaDB [cacti]> select * from poller_reindex;
+---------+---------------+--------+---------+----+--------------+--------------------+
| host_id | data_query_id | action | present | op | assert_value | arg1               |
+---------+---------------+--------+---------+----+--------------+--------------------+
|     154 |             1 |      0 |       1 | <  | 2668336600   | .1.3.6.1.2.1.1.3.0 |
|     172 |             1 |      0 |       1 | <  | 1512489000   | .1.3.6.1.2.1.1.3.0 |
|     174 |             1 |      0 |       1 | <  | 148659300    | .1.3.6.1.2.1.1.3.0 |
|     178 |             1 |      0 |       1 | <  | 1905199700   | .1.3.6.1.2.1.1.3.0 |
|     181 |             1 |      0 |       1 | <  | 657306700    | .1.3.6.1.2.1.1.3.0 |
|     181 |             4 |      0 |       1 | <  | 657306700    | .1.3.6.1.2.1.1.3.0 |
|     181 |             5 |      0 |       1 | <  | 657306700    | .1.3.6.1.2.1.1.3.0 |
|     191 |             1 |      0 |       1 | <  | 1053394400   | .1.3.6.1.2.1.1.3.0 |
|     196 |             1 |      0 |       1 | <  | 162434600    | .1.3.6.1.2.1.1.3.0 |
|     197 |             1 |      0 |       1 | <  | 163897200    | .1.3.6.1.2.1.1.3.0 |
|     208 |             1 |      0 |       1 | <  | 3485822400   | .1.3.6.1.2.1.1.3.0 |
|     210 |             1 |      0 |       1 | <  | 374328600    | .1.3.6.1.2.1.1.3.0 |
|     212 |             1 |      0 |       1 | <  | 1702366000   | .1.3.6.1.2.1.1.3.0 |
|     212 |            12 |      0 |       1 | <  | 1702366000   | .1.3.6.1.2.1.1.3.0 |
|     212 |            13 |      0 |       1 | <  | 1702366000   | .1.3.6.1.2.1.1.3.0 |
|     213 |             1 |      0 |       1 | <  | 1702602700   | .1.3.6.1.2.1.1.3.0 |
|     223 |            12 |      0 |       1 | <  | 1391451500   | .1.3.6.1.2.1.1.3.0 |
|     223 |            13 |      0 |       1 | <  | 1391451500   | .1.3.6.1.2.1.1.3.0 |
|     224 |             1 |      0 |       1 | <  | 1390979000   | .1.3.6.1.2.1.1.3.0 |
|     224 |            12 |      0 |       1 | <  | 1390979000   | .1.3.6.1.2.1.1.3.0 |
|     224 |            13 |      0 |       1 | <  | 1390979000   | .1.3.6.1.2.1.1.3.0 |
|     225 |             1 |      0 |       1 | <  | 1391262800   | .1.3.6.1.2.1.1.3.0 |
|     225 |            12 |      0 |       1 | <  | 1391262800   | .1.3.6.1.2.1.1.3.0 |
|     225 |            13 |      0 |       1 | <  | 1391262800   | .1.3.6.1.2.1.1.3.0 |
|     226 |             1 |      0 |       1 | <  | 1390886300   | .1.3.6.1.2.1.1.3.0 |
|     226 |            12 |      0 |       1 | <  | 1390886300   | .1.3.6.1.2.1.1.3.0 |
|     226 |            13 |      0 |       1 | <  | 1390886300   | .1.3.6.1.2.1.1.3.0 |
|     227 |             1 |      0 |       1 | <  | 1431584600   | .1.3.6.1.2.1.1.3.0 |
|     227 |            12 |      0 |       1 | <  | 1431584600   | .1.3.6.1.2.1.1.3.0 |
|     227 |            13 |      0 |       1 | <  | 1431584600   | .1.3.6.1.2.1.1.3.0 |
|     228 |             1 |      0 |       1 | <  | 1431313600   | .1.3.6.1.2.1.1.3.0 |
|     228 |            12 |      0 |       1 | <  | 1431313600   | .1.3.6.1.2.1.1.3.0 |
|     228 |            13 |      0 |       1 | <  | 1431313600   | .1.3.6.1.2.1.1.3.0 |
|     229 |             1 |      0 |       1 | <  | 1431614400   | .1.3.6.1.2.1.1.3.0 |
|     229 |            12 |      0 |       1 | <  | 1431614400   | .1.3.6.1.2.1.1.3.0 |
|     229 |            13 |      0 |       1 | <  | 1431614400   | .1.3.6.1.2.1.1.3.0 |
|     230 |             1 |      0 |       1 | <  | 1431311300   | .1.3.6.1.2.1.1.3.0 |
|     230 |            12 |      0 |       1 | <  | 1431311300   | .1.3.6.1.2.1.1.3.0 |
|     230 |            13 |      0 |       1 | <  | 1431311300   | .1.3.6.1.2.1.1.3.0 |
|     231 |             1 |      0 |       1 | <  | 1551906500   | .1.3.6.1.2.1.1.3.0 |
|     231 |            12 |      0 |       1 | <  | 1551906500   | .1.3.6.1.2.1.1.3.0 |
|     231 |            13 |      0 |       1 | <  | 1551906500   | .1.3.6.1.2.1.1.3.0 |
|     232 |             1 |      0 |       1 | <  | 1552112600   | .1.3.6.1.2.1.1.3.0 |
|     232 |            12 |      0 |       1 | <  | 1552112600   | .1.3.6.1.2.1.1.3.0 |
|     232 |            13 |      0 |       1 | <  | 1552112600   | .1.3.6.1.2.1.1.3.0 |
|     233 |             1 |      0 |       1 | <  | 234753000    | .1.3.6.1.2.1.1.3.0 |
|     234 |            12 |      0 |       1 | <  | 9096000      | .1.3.6.1.2.1.1.3.0 |
|     234 |            13 |      0 |       1 | <  | 9096000      | .1.3.6.1.2.1.1.3.0 |
|     238 |             1 |      0 |       1 | <  | 2666819000   | .1.3.6.1.2.1.1.3.0 |
|     241 |             1 |      0 |       1 | <  | 162434900    | .1.3.6.1.2.1.1.3.0 |
|     242 |             1 |      0 |       1 | <  | 373990800    | .1.3.6.1.2.1.1.3.0 |
|     245 |             1 |      0 |       1 | <  | 162441900    | .1.3.6.1.2.1.1.3.0 |
|     250 |             1 |      0 |       1 | <  | 653386400    | .1.3.6.1.2.1.1.3.0 |
|     261 |             1 |      0 |       1 | <  | 1225353400   | .1.3.6.1.2.1.1.3.0 |
|     261 |            12 |      0 |       1 | <  | 1225353400   | .1.3.6.1.2.1.1.3.0 |
|     273 |             1 |      0 |       1 | <  | 111366000    | .1.3.6.1.2.1.1.3.0 |
|     751 |             1 |      0 |       1 | <  | 1877593300   | .1.3.6.1.2.1.1.3.0 |
|     751 |             4 |      0 |       1 | <  | 1877593400   | .1.3.6.1.2.1.1.3.0 |
|     751 |             5 |      0 |       1 | <  | 1877593400   | .1.3.6.1.2.1.1.3.0 |
|     770 |             1 |      0 |       1 | <  | 584017000    | .1.3.6.1.2.1.1.3.0 |
|     770 |             4 |      0 |       1 | <  | 584017100    | .1.3.6.1.2.1.1.3.0 |
|     770 |             5 |      0 |       1 | <  | 584017100    | .1.3.6.1.2.1.1.3.0 |
|     777 |             1 |      0 |       1 | <  | 320391000    | .1.3.6.1.2.1.1.3.0 |
|     778 |             1 |      0 |       1 | <  | 391685300    | .1.3.6.1.2.1.1.3.0 |
|     797 |             1 |      0 |       1 | <  | 107348500    | .1.3.6.1.2.1.1.3.0 |
|    1793 |             1 |      0 |       1 | <  | 146905200    | .1.3.6.1.2.1.1.3.0 |
|    1793 |             4 |      0 |       1 | <  | 146905200    | .1.3.6.1.2.1.1.3.0 |
|    1793 |             5 |      0 |       1 | <  | 146905200    | .1.3.6.1.2.1.1.3.0 |
|    1806 |             1 |      0 |       1 | <  | 1981783200   | .1.3.6.1.2.1.1.3.0 |
|    1806 |             4 |      0 |       1 | <  | 1981783200   | .1.3.6.1.2.1.1.3.0 |
|    1806 |             5 |      0 |       1 | <  | 1981783200   | .1.3.6.1.2.1.1.3.0 |
|    1807 |             1 |      0 |       1 | <  | 1839411700   | .1.3.6.1.2.1.1.3.0 |
|    1807 |             4 |      0 |       1 | <  | 1839411700   | .1.3.6.1.2.1.1.3.0 |
|    1807 |             5 |      0 |       1 | <  | 1839411700   | .1.3.6.1.2.1.1.3.0 |
|    1813 |             1 |      0 |       1 | <  | 91940500     | .1.3.6.1.2.1.1.3.0 |
|    2037 |             1 |      0 |       1 | <  | 3518182900   | .1.3.6.1.2.1.1.3.0 |
|    2593 |             1 |      0 |       1 | <  | 3491575900   | .1.3.6.1.2.1.1.3.0 |
|    2596 |             1 |      0 |       1 | <  | 13891800     | .1.3.6.1.2.1.1.3.0 |
|    2596 |            12 |      0 |       1 | <  | 13891800     | .1.3.6.1.2.1.1.3.0 |
|    2596 |            13 |      0 |       1 | <  | 13891800     | .1.3.6.1.2.1.1.3.0 |
|    2753 |             1 |      0 |       1 | <  | 3518178000   | .1.3.6.1.2.1.1.3.0 |
|    2783 |             1 |      0 |       1 | <  | 1225161700   | .1.3.6.1.2.1.1.3.0 |
|    2783 |            12 |      0 |       1 | <  | 1225161800   | .1.3.6.1.2.1.1.3.0 |
|    2784 |             1 |      0 |       1 | <  | 1225160700   | .1.3.6.1.2.1.1.3.0 |
|    2784 |            12 |      0 |       1 | <  | 1225160800   | .1.3.6.1.2.1.1.3.0 |
|    2817 |             1 |      0 |       1 | <  | 197609000    | .1.3.6.1.2.1.1.3.0 |
|    2817 |            12 |      0 |       1 | <  | 197609100    | .1.3.6.1.2.1.1.3.0 |
|    2817 |            13 |      0 |       1 | <  | 197609100    | .1.3.6.1.2.1.1.3.0 |
+---------+---------------+--------+---------+----+--------------+--------------------+
88 rows in set (0.000 sec)

@jdcoats
Copy link
Author

jdcoats commented Oct 24, 2021

looks good now, its been 6 polling cycles and no more recache events

@TheWitness
Copy link
Member

Okay good. Thanks for being persistent.

@bmfmancini
Copy link
Member

bmfmancini commented Oct 24, 2021 via email

@jdcoats
Copy link
Author

jdcoats commented Oct 24, 2021 via email

@jdcoats jdcoats closed this as completed Oct 24, 2021
@github-actions github-actions bot locked and limited conversation to collaborators Jan 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Undesired behaviour resolved A fixed issue
Projects
None yet
Development

No branches or pull requests

3 participants