Error on interpretation of snmpUtime, when to big

[arno-st]

On a device when uptime is to long (more than 224 days I think) the device is seen as down, if the monitoring is set to snmpUptime and ping.

As an exemple I have this:
Uptime: 4190934469 (485days, 1hours, 29minutes)
and device is down.
ON 1.2.18 I have this:

sre-core	10.0.2.26	3159	28	29	Down	2d:23h:56m	N/A

And on 1.2.17

sre-core	10.0.2.26	363	37	38	Up	226d:23h:49m	485d:1h:55m

[netniV]

This may be the difference between 32-bit and 64-bit uptime counters. Uptime is recorded in seconds if I remember rightly as I used to have this issue on older 32-bit hardware making Cacti think it had rebooted when it hadn't.

What is your OS/Arch for all involved systems?

[arno-st]

My 1.2.18 run on centos 10.0-1160.42.2.el7.x86_64

My 1.2.17 run on centos 10.0-1160.42.2.el7.x86_64

The only point can find is that on has PHP 7.4.2 on the 1.2.17 version, the other one has 7.4.14 where cacti 1.2.18 run.

Otherwise, it should be same same!

As for the client I'm polling there are both the same device Cisco Switch

[netniV]

What about the SNMP libraries? Are you using php-snmp or net-snmp?

[arno-st]

php-snmp
both same version
Name : php-snmp
Arch : x86_64
Version : 7.4.24
Release : 1.el7.remi

[arno-st]

Don't think the problem is on the polling part.
since the DB give me both:
cacti 1.2.17 4165077143
cacti 1.2.18 4192683188

Mineur difference.

[TheWitness]

I think we need to move that column to a bigint later tonight my time unless @netniV want's to hammer it out.

[TheWitness]

Okay, this is resolved for the 1.2.19 release. You can just hand run the two SQL alters at the bottom of the 1_2_19.php file if you want to hack it in.

[TheWitness]

Thanks for keeping your eye on the ball.

[arno-st]

Can you reopen it, it dosen't solve the problem.
I'm working on it to find what's wrong.

Here is the ouput of my DB, the first 2 record are in SNMP v3 and are show as DOWN, the last one is in SNMP V2, and see as UP. I'm looking on that track to see if it's a SNMP version problem.
"id","poller_id","site_id","host_template_id","description","hostname","location","notes","external_id","snmp_community","snmp_version","snmp_username","snmp_password","snmp_auth_protocol","snmp_priv_passphrase","snmp_priv_protocol","snmp_context","snmp_engine_id","snmp_port","snmp_timeout","snmp_sysDescr","snmp_sysObjectID","snmp_sysUpTimeInstance","snmp_sysContact","snmp_sysName","snmp_sysLocation","availability_method","ping_method","ping_port","ping_timeout","ping_retries","max_oids","bulk_walk_size","device_threads","deleted","disabled","monitor","monitor_text","monitor_criticality","monitor_warn","monitor_alert","thold_send_email","thold_host_email","status","status_event_count","status_fail_date","status_rec_date","status_last_error","min_time","max_time","cur_time","avg_time","polling_time","total_polls","failed_polls","availability","last_updated","serial_no","model","isPhone","keep_mac_track","password","console_type","can_be_upgraded","can_be_rebooted","do_backup","login","mode"

"3159","1","3","8","core","10.0.2.26",," EZV: En service","SR02129 SR02128 ","telvlsn","3","SNMP_USER","SNMP_KEY","SHA","SNMP_KEY","AES128",,,"161","500","Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Thu 30-Jan-20 18:53 by mcpre","iso.3.6.1.4.1.9.1.2593","4165077143",,"CORE","LOCATION","1","1","23","400","1","10","-1","1",,,,,"0","0","0","1","0","1","5670","2021-10-15 10:06:02","2021-10-13 15:13:02","Device responded to SNMP, ICMP: Destination address not specified","0.00000","88.03201","0.00000","0.74397","0.027","795147","8664","98.91040","2021-10-19 08:35:02","FCW2211A0AT FCW2211A0B8","C9500-16X","of","of","SNMP_KEY","1","off","off","on","PASSWORD","bundle"

"3160","1","3","7","vbb","10.1.128.10",,,,"telvlsn","3","SNMP_USER","SNMP_KEY","SHA","SNMP_KEY","AES128",,,"161","500","Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.08.08.E RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Fri 1","iso.3.6.1.4.1.9.1.1732","2013484542","SOI - Telecom","VBB.recolte.lausanne.ch","LOCATION","1","1","23","400","1","10","-1","1",,,,,"0","0","0","1","0","1","11080","2021-10-11 15:56:02","2020-06-20 08:27:00","Device responded to SNMP, ICMP: Destination address not specified","0.03791","995.94558","1.81699","6.07636","0.034","795146","11232","98.58740","2021-10-19 08:35:02","JAE17430H3B JAE17430BD6","WS-C4500X-32","of","of","SNMP_KEY","1","off","off","on","PASSWORD","bundle"

"4515","1","3","9","pdp","10.128.1.41",," EZV: En service","SR01288 ","telvlsn","2",,,,,,,,"161","500","Cisco NX-OS(tm) n5000, Software (n5000-uk9), Version 5.1(3)N2(1b), RELEASE SOFTWARE Copyright (c) 2002-2011 by Cisco Systems, Inc. Device Manager Version 5.2(1), Compiled 8/31/2012 17:00:00","iso.3.6.1.4.1.9.12.3.1.3.1084","1643546154","","PDP","LOCATION","4","1","23","400","1","10","-1","1",,,"on",,"0","0","0","1","0","3","0","2021-08-04 18:05:05","2021-08-04 18:19:02","Device did not respond to SNMP, ICMP: Ping timed out","0.00000","55.65691","0.00000","1.20858","0.046","371268","14","99.99620","2021-10-19 08:35:02","SSI154300JT","N5K-C5548UP",,"of","UNAW2m3sFF+9uSzZf","1","off","off","off","PASSWORD","bundle"

[TheWitness]

Using spine or cmd.php? Problem could be spine at this point.

[arno-st]

I'm using spine, and I'm looking into it.
Don't know yet, I have tu understand how the status is check.
how can I debug spine ?
Does it affect all poller, or can I run it once for a specific device ?

[arno-st]

So here is the output of spine:
./spine -S -H 3160 --verbosity=5 --conf=../etc/spine.conf
SPINE: Using spine config file [../etc/spine.conf]
Total[0.0064] DEBUG: The path_php_server variable is /usr/share/cacti/script_server.php
Total[0.0065] DEBUG: The path_cactilog variable is /usr/share/cacti/log/cacti.log
Total[0.0065] DEBUG: The log_destination variable is 4 (STDOUT)
Total[0.0067] DEBUG: The path_php variable is /bin/php
Total[0.0069] DEBUG: The availability_method variable is 4
Total[0.0070] DEBUG: The ping_recovery_count variable is 6
Total[0.0071] DEBUG: The ping_failure_count variable is 4
Total[0.0072] DEBUG: The ping_method variable is 1
Total[0.0073] DEBUG: The ping_retries variable is 1
Total[0.0074] DEBUG: The ping_timeout variable is 400
Total[0.0074] DEBUG: The snmp_retries variable is 3
Total[0.0075] DEBUG: The log_perror variable is 1
Total[0.0076] DEBUG: The log_pwarn variable is 1
Total[0.0077] DEBUG: The boost_redirect variable is 1
Total[0.0078] DEBUG: The boost_rrd_update_enable variable is 0
Total[0.0079] DEBUG: The log_pstats variable is 1
Total[0.0080] DEBUG: The threads variable is 13
Total[0.0081] DEBUG: The polling interval is 60 seconds
Total[0.0082] DEBUG: The number of concurrent processes is 2
Total[0.0083] DEBUG: The script timeout is 25
Total[0.0084] DEBUG: The selective_device_debug variable is 3160,3159
Total[0.0085] DEBUG: The spine_log_level variable is 0
Total[0.0086] DEBUG: The number of php script servers to run is 5
Total[0.0087] DEBUG: Device List to be polled='3160', TotalPHPScripts='1'
Total[0.0087] DEBUG: The PHP Script Server is Required
Total[0.0088] DEBUG: The Maximum SNMP OID Get Size is 10
Total[0.0088] DEBUG: Selective Debug Devices 3160,3159
Total[0.0090] DEBUG: Total Connections made 1
Total[0.0090] DEBUG: Creating Local Connection Pool of 13 threads.
Total[0.0090] DEBUG: Creating Local Connection 0.
Total[0.0092] DEBUG: Total Connections made 2
Total[0.0096] DEBUG: Creating Local Connection 1.
Total[0.0098] DEBUG: Total Connections made 3
Total[0.0101] DEBUG: Creating Local Connection 2.
Total[0.0103] DEBUG: Total Connections made 4
Total[0.0107] DEBUG: Creating Local Connection 3.
Total[0.0109] DEBUG: Total Connections made 5
Total[0.0113] DEBUG: Creating Local Connection 4.
Total[0.0115] DEBUG: Total Connections made 6
Total[0.0119] DEBUG: Creating Local Connection 5.
Total[0.0120] DEBUG: Total Connections made 7
Total[0.0124] DEBUG: Creating Local Connection 6.
Total[0.0126] DEBUG: Total Connections made 8
Total[0.0130] DEBUG: Creating Local Connection 7.
Total[0.0132] DEBUG: Total Connections made 9
Total[0.0136] DEBUG: Creating Local Connection 8.
Total[0.0138] DEBUG: Total Connections made 10
Total[0.0142] DEBUG: Creating Local Connection 9.
Total[0.0143] DEBUG: Total Connections made 11
Total[0.0147] DEBUG: Creating Local Connection 10.
Total[0.0149] DEBUG: Total Connections made 12
Total[0.0153] DEBUG: Creating Local Connection 11.
Total[0.0155] DEBUG: Total Connections made 13
Total[0.0159] DEBUG: Creating Local Connection 12.
Total[0.0161] DEBUG: Total Connections made 14
Total[0.0166] DEBUG: Version 1.2.18 starting
Total[0.0166] DEBUG: MySQL is Thread Safe!
Total[0.0166] DEBUG: Spine running as 0 UID, 0 EUID
Total[0.0167] DEBUG: Spine is running as root.
Total[0.0167] DEBUG: Spine has got ICMP
Total[0.0167] DEBUG: Initializing Net-SNMP API
Total[0.0167] DEBUG: Issues with SNMP Header Version information, assuming old version of Net-SNMP.
Total[0.0180] DEBUG: Initializing PHP Script Server(s)
Total[0.0180] DEBUG: SS[0] PHP Script Server Routine Starting
Total[0.0180] DEBUG: SS[0] PHP Script Server About to FORK Child Process
Total[0.0185] DEBUG: SS[0] PHP Script Server Child FORK Success
Total[0.1668] DEBUG: SS[0] Confirmed PHP Script Server running using readfd[20], writefd[19]
Total[0.1668] DEBUG: SS[1] PHP Script Server Routine Starting
Total[0.1668] DEBUG: SS[1] PHP Script Server About to FORK Child Process
Total[0.1670] DEBUG: SS[1] PHP Script Server Child FORK Success
Total[0.3097] DEBUG: SS[1] Confirmed PHP Script Server running using readfd[22], writefd[21]
Total[0.3097] DEBUG: SS[2] PHP Script Server Routine Starting
Total[0.3097] DEBUG: SS[2] PHP Script Server About to FORK Child Process
Total[0.3099] DEBUG: SS[2] PHP Script Server Child FORK Success
Total[0.4525] DEBUG: SS[2] Confirmed PHP Script Server running using readfd[24], writefd[23]
Total[0.4525] DEBUG: SS[3] PHP Script Server Routine Starting
Total[0.4525] DEBUG: SS[3] PHP Script Server About to FORK Child Process
Total[0.4527] DEBUG: SS[3] PHP Script Server Child FORK Success
Total[0.6048] DEBUG: SS[3] Confirmed PHP Script Server running using readfd[26], writefd[25]
Total[0.6048] DEBUG: SS[4] PHP Script Server Routine Starting
Total[0.6049] DEBUG: SS[4] PHP Script Server About to FORK Child Process
Total[0.6050] DEBUG: SS[4] PHP Script Server Child FORK Success
Total[0.7482] DEBUG: SS[4] Confirmed PHP Script Server running using readfd[28], writefd[27]
Total[0.7492] Spine will support multithread device polling.
Total[0.7497] DEBUG: Initial Value of Available Threads is 13 (0 outstanding)
Total[0.7499] DEBUG: Valid Thread to be Created
Total[0.7500] DEBUG: Available Threads is 12 (1 outstanding)
Total[0.7500] DEBUG: In Poller, About to Start Polling of Device for Device ID 0
Total[0.7500] DEBUG: Traversing Local Connection Pool for free connection.
Total[0.7500] DEBUG: Checking Local Pool ID 0.
Total[0.7500] DEBUG: Allocating Local Pool ID 0.
Total[0.7502] DEBUG: Valid Thread to be Created
Total[0.7502] DEBUG: Available Threads is 11 (2 outstanding)
Total[0.7502] WARNING: Spine Sleeping While Waiting for 2 Threads to End
Total[0.7502] DEBUG: In Poller, About to Start Polling of Device for Device ID 3160
Total[0.7503] Device[0] HT[1] Updating Poller Items for Next Poll
Total[0.7503] DEBUG: Traversing Local Connection Pool for free connection.
Total[0.7503] DEBUG: Checking Local Pool ID 0.
Total[0.7503] DEBUG: Checking Local Pool ID 1.
Total[0.7503] DEBUG: Allocating Local Pool ID 1.
Total[0.7506] Device[0] HT[1] Total Time: 0.00064 Seconds
Total[0.7508] get_namebyhost(10.1.128.10) - Allocating name_t
Total[0.7508] get_namebyhost(10.1.128.10) - Token #1
Total[0.7508] get_hostbyname(10.1.128.10) - No matching method for 11 chars: 10.1.128.10
Total[0.7508] get_namebyhost(10.1.128.10) - Setting hostname: 10.1.128.10
Total[0.7508] DEBUG: Freeing Local Pool ID 0
Total[0.7508] DEBUG: Device[0] HT[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
Total[0.7509] Device[3160] INFO: SNMP Device '10.1.128.10' has timeout 500000 (500), retries 3
Total[0.7770] Device[3160] IPv4 address 10.1.128.10 (10.1.128.10)

Total[0.7770] Device[3160] DEBUG: Entering ICMP Ping
Total[0.7771] WARNING: Spine Sleeping While Waiting for 1 Threads to End
Total[0.7778] WARNING: Spine Sleeping While Waiting for 1 Threads to End
Total[0.7778] WARNING: Spine Sleeping While Waiting for 1 Threads to End
Total[0.7778] WARNING: Spine Sleeping While Waiting for 1 Threads to End
Total[0.7779] Device[3160] DEBUG: Entering SNMP Ping
Total[0.7823] Device[3160] PING Result: ICMP: Destination address not specified
Total[0.7823] Device[3160] SNMP Result: Device responded to SNMP

Total[0.7851] Device[3160] HT[1] NOTE: There are '350' Polling Items for this Device
Total[0.7852] DEBUG: Setting up writes to local database
Total[0.7854] Device[3160] HT[1] Updating Poller Items for Next Poll
Total[0.7865] Device[3160] HT[1] Total Time: 0.036 Seconds
Total[0.7867] DEBUG: Freeing Local Pool ID 1
Total[0.7867] DEBUG: Device[3160] HT[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
Total[1.2779] The Final Value of Threads is 0
Total[1.2786] DEBUG: Closing Local Connection Pool ID 0
Total[1.2786] DEBUG: Closing Local Connection Pool ID 1
Total[1.2787] DEBUG: Closing Local Connection Pool ID 2
Total[1.2787] DEBUG: Closing Local Connection Pool ID 3
Total[1.2787] DEBUG: Closing Local Connection Pool ID 4
Total[1.2787] DEBUG: Closing Local Connection Pool ID 5
Total[1.2787] DEBUG: Closing Local Connection Pool ID 6
Total[1.2787] DEBUG: Closing Local Connection Pool ID 7
Total[1.2787] DEBUG: Closing Local Connection Pool ID 8
Total[1.2787] DEBUG: Closing Local Connection Pool ID 9
Total[1.2788] DEBUG: Closing Local Connection Pool ID 10
Total[1.2788] DEBUG: Closing Local Connection Pool ID 11
Total[1.2788] DEBUG: Closing Local Connection Pool ID 12
Total[1.2788] DEBUG: Thread Cleanup Complete
Total[1.2788] DEBUG: SS[0] Script Server Shutdown Started
Total[1.3289] DEBUG: SS[1] Script Server Shutdown Started
Total[1.3790] DEBUG: SS[2] Script Server Shutdown Started
Total[1.4291] DEBUG: SS[3] Script Server Shutdown Started
Total[1.4793] DEBUG: SS[4] Script Server Shutdown Started
Total[1.5294] DEBUG: PHP Script Server Pipes Closed
Total[1.5294] DEBUG: Allocated Variable Memory Freed
Total[1.5294] DEBUG: MYSQL Free & Close Completed
Total[1.5295] DEBUG: Net-SNMP Close Completed
Total[1.5295] Time: 1.2779 s, Threads: 13, Devices: 2

[TheWitness]

Open a spine bug would you. Cacti is fixed.

[TheWitness]

Let me take that back, spine uses a string. So, it might have something to do with the snmp library. In the mean time. Edit poller.c and make the modification as in the highlighted row below:

Then, make spine and run as follows:

./spine -R --mibs --first host_id1 --last host_id2
<snip>
NOTE: The SNMP Uptime was 8536518
NOTE: The SNMP Uptime was 8537397
NOTE: The SNMP Uptime was 8536553
NOTE: The SNMP Uptime was 318654799
NOTE: The SNMP Uptime was 318654799
Time: 2.8684 s, Threads: 4, Devices: 47

Which should show the output above. Let us know if the value is correct there.

[TheWitness]

Continuing this discussion on the spine side.

[TheWitness]

Okay, made a few more GUI changes so that when you edit the device, you also see the correct uptime. Also addressed cmd.php and reindexing there.

[arno-st]

Just wondering where did you change the GUI

[TheWitness]

What do you mean? If you edit the device, it grabs uptime dynamically.

[arno-st]

Ho ok, I where wondering if you add a field.
Because the uptime was visible on the snmp Information, and didn't check the device page.