Repository navigation
Description
Hey All
I am having a weird issue and I see a descrpency between spine and netsnmp
I have some new devices in my lab around 400 of them I recently started noticing that a handful of them seem to always be set for recache
Digging furthure I see that the recache has been triggered because of .1.3.6.1.2.1.1.3.0 this is a wireless modem and the OID puts out the uptime of the wireless connection and not the modem system uptime for what ever reason
Here is the log
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[609] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[655] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[613] HT[1] DQ[14] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4129044 < output: 4135832)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[613] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4129044 < output: 4135832)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[613] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[611] HT[1] DQ[14] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 1416513 < output: 1423302)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[611] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 1416513 < output: 1423302)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[586] HT[1] DQ[14] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4112543 < output: 4119352)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[586] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4112543 < output: 4119352)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[611] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[586] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'
I also started seeing the following in the error log
2020-10-09 15:27:21 - POLLER: Poller[1] ERROR: Process being killed due to timeout! (commands, master, 1, 21242)
2020-10-09 15:27:19 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[642]
2020-10-09 15:27:19 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[670]
2020-10-09 15:27:19 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[694]
2020-10-09 15:27:18 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[661]
2020-10-09 15:27:18 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[611]
2020-10-09 15:27:17 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[664]
When I do a poll direct from spine I get the following
./spine -R -H 329 | more
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:29:35 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:29:35 - SPINE: Poller[1] PID[18512] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[329]
But if I do a snmpwalk on that OID it is responding fine
Spine v1.2.12
Cacti V1.2.12
NET-SNMP version: 5.7.2
Activity
bmfmancini commentedon Oct 9, 2020
I am trying this on cmd.php right now to see if the same behavior happens
bmfmancini commentedon Oct 9, 2020
Hmm same thing with cmd.php
bmfmancini commentedon Oct 9, 2020
I also found the following
Its seems spine is doing alot with this OID almost like a mini loop
TheWitness commentedon Oct 10, 2020
Disable data collector sync. We've found that this causes the issue.
TheWitness commentedon Oct 10, 2020
Might be something else too.
bmfmancini commentedon Oct 10, 2020
bmfmancini commentedon Oct 13, 2020
So deeper look it turns out its not just a handful of devices its all of this specific device but the error remains the same unable to fetch .1.3.6.1.2.1.1.3.0' snmpwalk still works fine on the same device fir that oid
[-]Spine V1.2.12 Recache due to failed to get OID but SNMPWALK works[/-][+]Recache due to failed to get OID but SNMPWALK works[/+]TheWitness commentedon Nov 21, 2020
Did this happen to happen when we switched from DST to ST?
TheWitness commentedon Nov 21, 2020
Nope:
2020: Sunday, March 8 and Sunday, November 1
TheWitness commentedon Nov 21, 2020
@bmfmancini, someone is messing with the clocks on those devices. That's the reason.
bmfmancini commentedon Nov 21, 2020
TheWitness commentedon Nov 21, 2020
What is the re-index method?
bmfmancini commentedon Nov 21, 2020
28 remaining items
TheWitness commentedon Nov 26, 2020
Yea, just about to make an update.
Fixing Issue #3931
TheWitness commentedon Nov 26, 2020
Okay, should be fixed now.
bmfmancini commentedon Nov 26, 2020
Ok testing now
TheWitness commentedon Nov 27, 2020
Bump!
bmfmancini commentedon Nov 27, 2020
TheWitness commentedon Nov 27, 2020
Okay, take this watch and re-write it for your system, and then run it. While it's running note the "assert" values. Then, do a full sync to the remote, the value on the remote should go back.
Output should look something like:
TheWitness commentedon Nov 27, 2020
You should notice that the Main collectors time should be back from before the last sync, and it should not be updated afterwards. The only person writing that assert_value will be the remote data collector unless you have another collector pushing data into your production system, which would be a setup error I think. For that, it might be good to log the connection attempts and where they are coming from.
TheWitness commentedon Nov 27, 2020
If the later is the case, you might want to consider a more strict ACL on who can connect to what databases, if you don't have that already. I don't want to jump to any conclusion here though.
bmfmancini commentedon Nov 27, 2020
I am trying this now the thing is I have devices doing this that are on the Main poller
TheWitness commentedon Nov 27, 2020
That's very odd then. Might even be some device issue.
[-]Recache due to failed to get OID but SNMPWALK works[/-][+]When syncing data collectors, a reindex event may be triggered unnecessarily[/+]TheWitness commentedon Dec 4, 2020
@bmfmancini, I'm marking this one closed as it was a real problem for remote data collectors, and that issue is resolved. As far as main data collector based devices, I suggest a zone collision or some hardware issue.