When syncing data collectors, a reindex event may be triggered unnecessarily #3931

bmfmancini · 2020-10-09T19:33:53Z

Hey All

I am having a weird issue and I see a descrpency between spine and netsnmp
I have some new devices in my lab around 400 of them I recently started noticing that a handful of them seem to always be set for recache

Digging furthure I see that the recache has been triggered because of .1.3.6.1.2.1.1.3.0 this is a wireless modem and the OID puts out the uptime of the wireless connection and not the modem system uptime for what ever reason

Here is the log

2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[609] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[655] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[613] HT[1] DQ[14] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4129044 < output: 4135832)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[613] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4129044 < output: 4135832)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[613] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[611] HT[1] DQ[14] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 1416513 < output: 1423302)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[611] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 1416513 < output: 1423302)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[586] HT[1] DQ[14] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4112543 < output: 4119352)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[586] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4112543 < output: 4119352)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[611] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[586] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'

I also started seeing the following in the error log

2020-10-09 15:27:21 - POLLER: Poller[1] ERROR: Process being killed due to timeout! (commands, master, 1, 21242)
2020-10-09 15:27:19 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[642]
2020-10-09 15:27:19 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[670]
2020-10-09 15:27:19 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[694]
2020-10-09 15:27:18 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[661]
2020-10-09 15:27:18 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[611]
2020-10-09 15:27:17 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[664]

When I do a poll direct from spine I get the following

./spine -R -H 329 | more
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:29:35 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:29:35 - SPINE: Poller[1] PID[18512] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[329]

But if I do a snmpwalk on that OID it is responding fine

Spine v1.2.12
Cacti V1.2.12
NET-SNMP version: 5.7.2

The text was updated successfully, but these errors were encountered:

bmfmancini · 2020-10-09T19:35:31Z

I am trying this on cmd.php right now to see if the same behavior happens

bmfmancini · 2020-10-09T19:42:08Z

Hmm same thing with cmd.php

bmfmancini · 2020-10-09T19:51:17Z

I also found the following

2020-10-09 15:49:25 - PCOMMAND Device[489] WARNING: Recache Event Detected for Device
2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.6.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.6.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.6.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.6.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.6.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.6.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.6.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.6.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.5.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.5.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.5.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.5.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.5.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.5.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.5.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.5.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.4.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.4.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.4.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.4.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.4.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.4.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.4.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.4.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.2.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.2.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.2.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.2.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.2.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.2.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.2.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.2.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.1.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.1.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.1.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.1.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.1.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.1.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.1.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.1.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0)

Its seems spine is doing alot with this OID almost like a mini loop

TheWitness · 2020-10-10T10:40:36Z

Disable data collector sync. We've found that this causes the issue.

TheWitness · 2020-10-10T10:41:36Z

Might be something else too.

bmfmancini · 2020-10-10T15:38:02Z

Just tried to disable replication but no change I am not sure if this is something with the device or Cacti at this point but as I say that I only see this happening on a hand full of the devices I have 400 of them in total but 40 of them are doing this

…

On Sat, Oct 10, 2020 at 6:41 AM TheWitness ***@***.***> wrote: Might be something else too. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/Cacti/spine/issues/173#issuecomment-706528464>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGEXTDBZ7QF572ESBF5JFLSKA26ZANCNFSM4SKOTC4A> .

-- Thank you Sean Mancini,LBBIT Owner/Principal Engineer www.seanmancini.com “Companies spend millions of dollars on firewalls, encryption, and secure access devices, and it’s money wasted because none of these measures address the weakest link in the security chain.” *– Kevin Mitnick*

bmfmancini · 2020-10-13T19:32:16Z

So deeper look it turns out its not just a handful of devices its all of this specific device but the error remains the same unable to fetch .1.3.6.1.2.1.1.3.0' snmpwalk still works fine on the same device fir that oid

TheWitness · 2020-11-21T00:28:37Z

Did this happen to happen when we switched from DST to ST?

TheWitness · 2020-11-21T00:32:23Z

Nope:

2020: Sunday, March 8 and Sunday, November 1

TheWitness · 2020-11-21T00:34:25Z

@bmfmancini, someone is messing with the clocks on those devices. That's the reason.

bmfmancini · 2020-11-21T03:01:14Z

That's weird it's happening at every poll would that mean the poller is seeing the time change every time ?

…

On Fri, Nov 20, 2020, 19:34 TheWitness ***@***.***> wrote: Closed Cacti/spine#173 <https://github.com/Cacti/spine/issues/173>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/Cacti/spine/issues/173#event-4023252563>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGEXTEYPACRBOGEBR7KJPDSQ4DK3ANCNFSM4SKOTC4A> .

TheWitness · 2020-11-21T03:24:12Z

What is the re-index method?

bmfmancini · 2020-11-21T03:33:43Z

It's set to uptime

…

On Fri, Nov 20, 2020, 22:24 TheWitness ***@***.***> wrote: What is the re-index method? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/Cacti/spine/issues/173#issuecomment-731500193>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGEXTCMBZ4B7UH3DRRSWVDSQ4XGRANCNFSM4SKOTC4A> .

TheWitness · 2020-11-21T03:39:56Z

Okay, so each polling cycle, Cacti takes the value from poller_reindex, the "assert_value" and compares it against the "arg1" snmpget of that value, and if the operator is wrong, then it causes that error. From my system below. Tinker with calling snmpget with the oid and what is stored in that table. These are also remote devices right?

TheWitness · 2020-11-21T03:40:12Z

This could be an artifact from a recent change BTW.

TheWitness · 2020-11-21T03:43:33Z

Okay, there is a problem. Lucky day!

TheWitness · 2020-11-21T03:51:51Z

Should have the solution shortly.

TheWitness · 2020-11-21T05:31:58Z

Turns out this is a Cacti issue. Committing in a bit.

- Recache due to failed to get OID but SNMPWALK works - Certain Device actions cause the removal of poller items from the remote data collector

TheWitness · 2020-11-21T05:37:58Z

Test ASAP. This will force us to move ahead the release.

bmfmancini · 2020-11-21T13:22:57Z

Thanks Larry I should be able to test this on monday if I get a chance earlier is I'll report back

…

On Sat, Nov 21, 2020, 00:38 TheWitness ***@***.***> wrote: Test ASAP. This will force us to move ahead the release. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3931 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGEXTHQ2MDKRXKNCMUR6ODSQ5G4FANCNFSM4T5SEDDQ> .

TheWitness · 2020-11-21T15:28:21Z

Sooner the better, I don't want this bug hanging out there for too long.

bmfmancini · 2020-11-23T14:46:07Z

@TheWitness

I tested this morning I grabbed the files off the 1.2.x branch same behavior is seen

bmfmancini · 2020-11-23T14:47:16Z

Should I try to rebuild the poller cache?

TheWitness · 2020-11-24T03:41:08Z

Yes.

bmfmancini · 2020-11-24T14:21:52Z

Ok just rebuilt the cache will report back soon

bmfmancini · 2020-11-24T19:44:01Z

@TheWitness No change on my side

TheWitness · 2020-11-25T14:11:45Z

Are your PHP file replicating to the remotes?

bmfmancini · 2020-11-25T14:15:07Z

I will confirm for sure but I am pretty sure they did

…

On Wed, Nov 25, 2020, 09:12 TheWitness ***@***.***> wrote: Are your PHP file replicating to the remotes? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3931 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGEXTGDX5RDCV66RP7ESIDSRUGDFANCNFSM4T5SEDDQ> .

TheWitness · 2020-11-25T19:00:09Z

It was buggered up on my system again. Research tonight.

bmfmancini · 2020-11-25T19:03:27Z

Ok cool thanks !

…

On Wed, Nov 25, 2020, 14:00 TheWitness ***@***.***> wrote: It was buggered up on my system again. Research tonight. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3931 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGEXTEV3IJBMJIF5IXEQ4LSRVH4RANCNFSM4T5SEDDQ> .

TheWitness · 2020-11-25T22:56:01Z

Okay, take the latest lib/poller.php and do a full sync to your pollers. This should get it fixed.

TheWitness · 2020-11-25T22:57:15Z

Related to #3916 and #3915

bmfmancini · 2020-11-26T15:06:09Z

Sorry man no Dice still coming in

TheWitness · 2020-11-26T15:16:36Z

So, the re-index warnings happen after replication still then? I want to ensure that we are not mixing issues. The poller cache evaporating vs. reindex errors.

bmfmancini · 2020-11-26T15:22:49Z

recache OID warnings still come even after replication

TheWitness · 2020-11-26T15:30:53Z

Yea, just about to make an update.

* forgot to handle the poller_reindex cache

TheWitness · 2020-11-26T15:53:47Z

Okay, should be fixed now.

bmfmancini · 2020-11-26T16:03:54Z

Ok testing now

TheWitness · 2020-11-27T15:05:52Z

Bump!

bmfmancini · 2020-11-27T15:12:17Z

Sorry I thought I replied Sorry man I'm still seeing the same thing

…

On Fri, Nov 27, 2020, 10:06 TheWitness ***@***.***> wrote: Bump! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3931 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGEXTCAVUHULORQBR7EOODSR6555ANCNFSM4T5SEDDQ> .

TheWitness · 2020-11-27T15:30:54Z

Okay, take this watch and re-write it for your system, and then run it. While it's running note the "assert" values. Then, do a full sync to the remote, the value on the remote should go back.

 watch 'echo "Main Hosts";mysql -e "select * from poller_reindex where host_id=13" cacti;echo "Remote Host";mysql -ucactiuser -pcactiuser -hvmhost1 -e "select * from poller_reindex where host_id=13" cacti';

Output should look something like:

Every 2.0s: echo "Main Hosts";mysql -e "select * from poller_reindex where host_id=13" cacti;echo "Remote Host";mysql -u...  Fri Nov 27 10:30:39 2020

Main Hosts
host_id data_query_id   action  present op      assert_value    arg1
13      4       0       1       =       6       .1.3.6.1.2.1.2.1.0
13      6       0       1       <       590286159       .1.3.6.1.2.1.1.3.0
13      7       0       1       <       633714161       .1.3.6.1.2.1.1.3.0
13      8       0       1       <       633714235       .1.3.6.1.2.1.1.3.0
13      9       0       1       <       633714273       .1.3.6.1.2.1.1.3.0
Remote Host
host_id data_query_id   action  present op      assert_value    arg1
13      4       0       1       =       6       .1.3.6.1.2.1.2.1.0
13      6       0       1       <       639166433       .1.3.6.1.2.1.1.3.0
13      7       0       1       <       639166435       .1.3.6.1.2.1.1.3.0
13      8       0       1       <       639166437       .1.3.6.1.2.1.1.3.0
13      9       0       1       <       639166439       .1.3.6.1.2.1.1.3.0

TheWitness · 2020-11-27T15:32:44Z

You should notice that the Main collectors time should be back from before the last sync, and it should not be updated afterwards. The only person writing that assert_value will be the remote data collector unless you have another collector pushing data into your production system, which would be a setup error I think. For that, it might be good to log the connection attempts and where they are coming from.

TheWitness · 2020-11-27T15:34:49Z

If the later is the case, you might want to consider a more strict ACL on who can connect to what databases, if you don't have that already. I don't want to jump to any conclusion here though.

bmfmancini · 2020-11-27T17:50:55Z

I am trying this now the thing is I have devices doing this that are on the Main poller

TheWitness · 2020-11-27T18:04:12Z

That's very odd then. Might even be some device issue.

TheWitness · 2020-12-04T22:26:32Z

@bmfmancini, I'm marking this one closed as it was a real problem for remote data collectors, and that issue is resolved. As far as main data collector based devices, I suggest a zone collision or some hardware issue.

TheWitness changed the title ~~Spine V1.2.12 Recache due to failed to get OID but SNMPWALK works~~ Recache due to failed to get OID but SNMPWALK works Nov 21, 2020

TheWitness closed this as completed Nov 21, 2020

TheWitness reopened this Nov 21, 2020

TheWitness transferred this issue from Cacti/spine Nov 21, 2020

TheWitness added a commit that referenced this issue Nov 21, 2020

Fixing Issue #3931, #3916

93dc93c

- Recache due to failed to get OID but SNMPWALK works - Certain Device actions cause the removal of poller items from the remote data collector

TheWitness added bug Undesired behaviour resolved A fixed issue labels Nov 21, 2020

TheWitness added this to the v1.2.16 milestone Nov 21, 2020

TheWitness added a commit that referenced this issue Nov 26, 2020

Fixing Issue #3931

f39e84f

* forgot to handle the poller_reindex cache

netniV changed the title ~~Recache due to failed to get OID but SNMPWALK works~~ When syncing data collectors, a reindex event may be triggered unnecessarily Nov 30, 2020

TheWitness closed this as completed Dec 4, 2020

github-actions bot locked and limited conversation to collaborators Mar 5, 2021

When syncing data collectors, a reindex event may be triggered unnecessarily #3931

When syncing data collectors, a reindex event may be triggered unnecessarily #3931

Comments

bmfmancini commented Oct 9, 2020

bmfmancini commented Oct 9, 2020

bmfmancini commented Oct 9, 2020

bmfmancini commented Oct 9, 2020

TheWitness commented Oct 10, 2020

TheWitness commented Oct 10, 2020

bmfmancini commented Oct 10, 2020 via email

bmfmancini commented Oct 13, 2020

TheWitness commented Nov 21, 2020

TheWitness commented Nov 21, 2020

TheWitness commented Nov 21, 2020

bmfmancini commented Nov 21, 2020 via email

TheWitness commented Nov 21, 2020

bmfmancini commented Nov 21, 2020 via email

TheWitness commented Nov 21, 2020

TheWitness commented Nov 21, 2020

TheWitness commented Nov 21, 2020

TheWitness commented Nov 21, 2020

TheWitness commented Nov 21, 2020

TheWitness commented Nov 21, 2020

bmfmancini commented Nov 21, 2020 via email

TheWitness commented Nov 21, 2020

bmfmancini commented Nov 23, 2020

bmfmancini commented Nov 23, 2020

TheWitness commented Nov 24, 2020

bmfmancini commented Nov 24, 2020

bmfmancini commented Nov 24, 2020

TheWitness commented Nov 25, 2020

bmfmancini commented Nov 25, 2020 via email

TheWitness commented Nov 25, 2020

bmfmancini commented Nov 25, 2020 via email

TheWitness commented Nov 25, 2020

TheWitness commented Nov 25, 2020

bmfmancini commented Nov 26, 2020

TheWitness commented Nov 26, 2020 • edited

bmfmancini commented Nov 26, 2020

TheWitness commented Nov 26, 2020

TheWitness commented Nov 26, 2020

bmfmancini commented Nov 26, 2020

TheWitness commented Nov 27, 2020

bmfmancini commented Nov 27, 2020 via email

TheWitness commented Nov 27, 2020

TheWitness commented Nov 27, 2020

TheWitness commented Nov 27, 2020

bmfmancini commented Nov 27, 2020

TheWitness commented Nov 27, 2020

TheWitness commented Dec 4, 2020

TheWitness commented Nov 26, 2020 •

edited