Skip to content

When syncing data collectors, a reindex event may be triggered unnecessarily #3931

Closed

Description

@bmfmancini

Hey All

I am having a weird issue and I see a descrpency between spine and netsnmp
I have some new devices in my lab around 400 of them I recently started noticing that a handful of them seem to always be set for recache

Digging furthure I see that the recache has been triggered because of .1.3.6.1.2.1.1.3.0 this is a wireless modem and the OID puts out the uptime of the wireless connection and not the modem system uptime for what ever reason

Here is the log

2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[609] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[655] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[613] HT[1] DQ[14] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4129044 < output: 4135832)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[613] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4129044 < output: 4135832)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[613] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[611] HT[1] DQ[14] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 1416513 < output: 1423302)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[611] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 1416513 < output: 1423302)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[586] HT[1] DQ[14] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4112543 < output: 4119352)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[586] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4112543 < output: 4119352)
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[611] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'
2020-10-09 15:07:19 - SPINE: Poller[1] PID[4082] Device[586] HT[1] RECACHE: Processing 2 items in the auto reindex cache for 'IP'

I also started seeing the following in the error log

2020-10-09 15:27:21 - POLLER: Poller[1] ERROR: Process being killed due to timeout! (commands, master, 1, 21242)
2020-10-09 15:27:19 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[642]
2020-10-09 15:27:19 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[670]
2020-10-09 15:27:19 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[694]
2020-10-09 15:27:18 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[661]
2020-10-09 15:27:18 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[611]
2020-10-09 15:27:17 - SPINE: Poller[1] PID[8933] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[664]

When I do a poll direct from spine I get the following

./spine -R -H 329 | more
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:29:33 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0)
2020-10-09 15:29:35 - SPINE: Poller[1] PID[18512] Device[329] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0) [complete]
2020-10-09 15:29:35 - SPINE: Poller[1] PID[18512] ERROR: Failed to get oid '.1.3.6.1.2.1.1.3.0' for Device[329]

But if I do a snmpwalk on that OID it is responding fine

Spine v1.2.12
Cacti V1.2.12
NET-SNMP version: 5.7.2

Activity

  1. bmfmancini commented on Oct 9, 2020

    @bmfmancini
    MemberAuthor

    I am trying this on cmd.php right now to see if the same behavior happens

  2. bmfmancini commented on Oct 9, 2020

    @bmfmancini
    MemberAuthor

    Hmm same thing with cmd.php

  3. bmfmancini commented on Oct 9, 2020

    @bmfmancini
    MemberAuthor

    I also found the following

    2020-10-09 15:49:25 - PCOMMAND Device[489] WARNING: Recache Event Detected for Device
    2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0) [complete]
    2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0)
    2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0) [complete]
    2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0)
    2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0) [complete]
    2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0)
    2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0) [complete]
    2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0)
    2020-10-09 15:49:04 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.6.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.6.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.6.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.6.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.6.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.6.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.6.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.6.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.5.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.5.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.5.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.5.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.5.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.5.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.5.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.5.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.4.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.4.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.4.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.4.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.4.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.4.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.4.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.4.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.2.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.2.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.2.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.2.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.2.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.2.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.2.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.2.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.1.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.1.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.1.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.1.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.1.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.1.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.1.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.1.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_sess_sync_response(.1.3.6.1.2.1.1.3.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_add_null_var(.1.3.6.1.2.1.1.3.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_parse_oid(.1.3.6.1.2.1.1.3.0)
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0) [complete]
    2020-10-09 15:49:03 - SPINE: Poller[1] PID[7263] Device[489] WARNING: snmp_pdu_create(.1.3.6.1.2.1.1.3.0)
    

    Its seems spine is doing alot with this OID almost like a mini loop

  4. TheWitness commented on Oct 10, 2020

    @TheWitness
    Member

    Disable data collector sync. We've found that this causes the issue.

  5. TheWitness commented on Oct 10, 2020

    @TheWitness
    Member

    Might be something else too.

  6. bmfmancini commented on Oct 10, 2020

    @bmfmancini
    MemberAuthor
  7. bmfmancini commented on Oct 13, 2020

    @bmfmancini
    MemberAuthor

    So deeper look it turns out its not just a handful of devices its all of this specific device but the error remains the same unable to fetch .1.3.6.1.2.1.1.3.0' snmpwalk still works fine on the same device fir that oid

  8. changed the title [-]Spine V1.2.12 Recache due to failed to get OID but SNMPWALK works[/-] [+]Recache due to failed to get OID but SNMPWALK works[/+] on Nov 21, 2020
  9. TheWitness commented on Nov 21, 2020

    @TheWitness
    Member

    Did this happen to happen when we switched from DST to ST?

  10. TheWitness commented on Nov 21, 2020

    @TheWitness
    Member

    Nope:

    2020: Sunday, March 8 and Sunday, November 1

  11. TheWitness commented on Nov 21, 2020

    @TheWitness
    Member

    @bmfmancini, someone is messing with the clocks on those devices. That's the reason.

  12. bmfmancini commented on Nov 21, 2020

    @bmfmancini
    MemberAuthor
  13. TheWitness commented on Nov 21, 2020

    @TheWitness
    Member

    What is the re-index method?

  14. bmfmancini commented on Nov 21, 2020

    @bmfmancini
    MemberAuthor
  15. 28 remaining items

  16. TheWitness commented on Nov 26, 2020

    @TheWitness
    Member

    Yea, just about to make an update.

  17. added a commit that references this issue on Nov 26, 2020
  18. TheWitness commented on Nov 26, 2020

    @TheWitness
    Member

    Okay, should be fixed now.

  19. bmfmancini commented on Nov 26, 2020

    @bmfmancini
    MemberAuthor

    Ok testing now

  20. TheWitness commented on Nov 27, 2020

    @TheWitness
    Member

    Bump!

  21. bmfmancini commented on Nov 27, 2020

    @bmfmancini
    MemberAuthor
  22. TheWitness commented on Nov 27, 2020

    @TheWitness
    Member

    Okay, take this watch and re-write it for your system, and then run it. While it's running note the "assert" values. Then, do a full sync to the remote, the value on the remote should go back.

     watch 'echo "Main Hosts";mysql -e "select * from poller_reindex where host_id=13" cacti;echo "Remote Host";mysql -ucactiuser -pcactiuser -hvmhost1 -e "select * from poller_reindex where host_id=13" cacti';

    Output should look something like:

    Every 2.0s: echo "Main Hosts";mysql -e "select * from poller_reindex where host_id=13" cacti;echo "Remote Host";mysql -u...  Fri Nov 27 10:30:39 2020
    
    Main Hosts
    host_id data_query_id   action  present op      assert_value    arg1
    13      4       0       1       =       6       .1.3.6.1.2.1.2.1.0
    13      6       0       1       <       590286159       .1.3.6.1.2.1.1.3.0
    13      7       0       1       <       633714161       .1.3.6.1.2.1.1.3.0
    13      8       0       1       <       633714235       .1.3.6.1.2.1.1.3.0
    13      9       0       1       <       633714273       .1.3.6.1.2.1.1.3.0
    Remote Host
    host_id data_query_id   action  present op      assert_value    arg1
    13      4       0       1       =       6       .1.3.6.1.2.1.2.1.0
    13      6       0       1       <       639166433       .1.3.6.1.2.1.1.3.0
    13      7       0       1       <       639166435       .1.3.6.1.2.1.1.3.0
    13      8       0       1       <       639166437       .1.3.6.1.2.1.1.3.0
    13      9       0       1       <       639166439       .1.3.6.1.2.1.1.3.0
    
  23. TheWitness commented on Nov 27, 2020

    @TheWitness
    Member

    You should notice that the Main collectors time should be back from before the last sync, and it should not be updated afterwards. The only person writing that assert_value will be the remote data collector unless you have another collector pushing data into your production system, which would be a setup error I think. For that, it might be good to log the connection attempts and where they are coming from.

  24. TheWitness commented on Nov 27, 2020

    @TheWitness
    Member

    If the later is the case, you might want to consider a more strict ACL on who can connect to what databases, if you don't have that already. I don't want to jump to any conclusion here though.

  25. bmfmancini commented on Nov 27, 2020

    @bmfmancini
    MemberAuthor

    I am trying this now the thing is I have devices doing this that are on the Main poller

  26. TheWitness commented on Nov 27, 2020

    @TheWitness
    Member

    That's very odd then. Might even be some device issue.

  27. changed the title [-]Recache due to failed to get OID but SNMPWALK works[/-] [+]When syncing data collectors, a reindex event may be triggered unnecessarily[/+] on Nov 30, 2020
  28. TheWitness commented on Dec 4, 2020

    @TheWitness
    Member

    @bmfmancini, I'm marking this one closed as it was a real problem for remote data collectors, and that issue is resolved. As far as main data collector based devices, I suggest a zone collision or some hardware issue.

  29. locked and limited conversation to collaborators on Mar 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugUndesired behaviourresolvedA fixed issue

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions