Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updating a Data Template does not switch rrd_heartbeat properly for all sources leading to empty graphs #4991

Closed
bmfmancini opened this issue Nov 7, 2022 · 7 comments
Closed

Updating a Data Template does not switch rrd_heartbeat properly for all sources leading to empty graphs #4991

bmfmancini opened this issue Nov 7, 2022 · 7 comments
Labels
bug Undesired behaviour confirmed Bug is confirm by dev team resolved A fixed issue
Milestone
v1.2.23

Comments

@bmfmancini
Copy link
Member

bmfmancini commented Nov 7, 2022 •
edited
Loading

Hey Guys,

I am seeing a weird issue with the following

Cisco ASA using the ASA template
SNMP is showing up and spine collects data but nothing plots on any snmp graph
Only graphs that work as the snmp uptime, polling time and ping
I have the same device on the same cacti version but polling every minute working without issue

image

Here is the Spine Data

2022-11-07 12:09:21 - SPINE: Poller[1] PID[619685] PT[140299301230336] Device[17147] HT[1] DS[549984] TT[14.80] SNMP: v3: (device_ip), dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.10, value: 1792801
2022-11-07 12:09:21 - SPINE: Poller[1] PID[619685] PT[140299301230336] Device[17147] HT[1] DS[549984] TT[14.81] SNMP: v3: (device_ip), dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.10, value: 0
2022-11-07 12:09:21 - SPINE: Poller[1] PID[619685] PT[140299301230336] Device[17147] HT[1] DS[549984] TT[14.82] SNMP: v3: (device_ip), dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.10, value: 0
2022-11-07 12:09:21 - SPINE: Poller[1] PID[619685] PT[140299301230336] Device[17147] HT[1] DS[549984] TT[14.83] SNMP: v3: (device_ip), dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.10, value: 0

Here is the Data source debug output shows that value is NaN

image

the data template is set to 5Min polling as it should be

image

Poller cache is showing the proper OID's

ASA test - Errors - GigabitEthernet0/7	ASA test	SNMP Version: 3, User: (snmp_user), OID: .1.3.6.1.2.1.2.2.1.13.10
RRD: /var/www/html/(path)/rra/asa_test_discards_in_549984.rrd
ASA test - Errors - GigabitEthernet0/7	ASA test	SNMP Version: 3, User: (snmp_user), OID: .1.3.6.1.2.1.2.2.1.19.10
RRD: /var/www/html/(path)/rra/asa_test_discards_in_549984.rrd
ASA test - Errors - GigabitEthernet0/7	ASA test	SNMP Version: 3, User: (snmp_user), OID: .1.3.6.1.2.1.2.2.1.14.10
RRD: /var/www/html/(path)/rra/asa_test_discards_in_549984.rrd
ASA test - Errors - GigabitEthernet0/7	ASA test	SNMP Version: 3, User: (snmp_user), OID: .1.3.6.1.2.1.2.2.1.20.10
RRD: /var/www/html/(path)/rra/asa_test_discards_in_549984.rrd

PHP-snmp is not installed

image

RRD permission's are correct

-rw-r--r--  1 apache apache 388K Nov  7 11:58 asa_test_5min_cpu_549976.rrd
-rw-r--r--  1 apache apache 1.6M Nov  7 11:58 asa_test_discards_in_549984.rrd
-rw-r--r--  1 apache apache 1.6M Nov  7 11:58 asa_test_discards_in_549985.rrd
-rw-r--r--  1 apache apache 1.6M Nov  7 11:58 asa_test_discards_in_549986.rrd
-rw-r--r--  1 apache apache 1.6M Nov  7 11:58 asa_test_discards_in_549987.rrd
-rw-r--r--  1 apache apache 1.6M Nov  7 11:58 asa_test_discards_in_549988.rrd
-rw-r--r--  1 apache apache 388K Nov  7 11:58 asa_test_polling_time_549977.rrd
-rw-r--r--  1 apache apache 774K Nov  7 11:58 asa_test_traffic_in_549979.rrd
-rw-r--r--  1 apache apache 774K Nov  7 11:58 asa_test_traffic_in_549980.rrd
-rw-r--r--  1 apache apache 774K Nov  7 11:58 asa_test_traffic_in_549981.rrd
-rw-r--r--  1 apache apache 774K Nov  7 11:58 asa_test_traffic_in_549982.rrd
-rw-r--r--  1 apache apache 774K Nov  7 11:58 asa_test_traffic_in_549983.rrd
-rw-r--r--  1 apache apache 388K Nov  7 11:58 asa_test_uptime_549978.rrd

RRD's are set to the correct step

filename = "asa_test_discards_in_549984.rrd"
rrd_version = "0003"
step = 300
last_update = 1667839887
header_size = 8240

Boost has the collected results waiting to be written

image

MariaDB [cacti]> select * from poller_output_boost where local_data_id = 549984;
+---------------+--------------+---------------------+---------+
| local_data_id | rrd_name     | time                | output  |
+---------------+--------------+---------------------+---------+
|        549984 | discards_in  | 2022-11-07 11:56:26 | 0       |
|        549984 | discards_out | 2022-11-07 11:56:26 | 0       |
|        549984 | errors_in    | 2022-11-07 11:56:26 | 1792801 |
|        549984 | errors_out   | 2022-11-07 11:56:26 | 0       |
|        549984 | discards_in  | 2022-11-07 12:01:29 | 0       |
|        549984 | discards_out | 2022-11-07 12:01:29 | 0       |
|        549984 | errors_in    | 2022-11-07 12:01:29 | 1792801 |
|        549984 | errors_out   | 2022-11-07 12:01:29 | 0       |
|        549984 | discards_in  | 2022-11-07 12:06:29 | 0       |
|        549984 | discards_out | 2022-11-07 12:06:29 | 0       |
|        549984 | errors_in    | 2022-11-07 12:06:29 | 1792801 |
@bmfmancini bmfmancini added bug Undesired behaviour unverified Some days we don't have a clue labels Nov 7, 2022
@bmfmancini
Copy link
Member Author

the troubleshooter seems to think everything is fine

image

image

@bmfmancini
Copy link
Member Author

Tried another ASA with snmpv2 no difference

@TheWitness
Copy link
Member

Post the RRDtool info for one of the RRDfiles.

@bmfmancini
Copy link
Member Author 

84.rrd
filename = "/var/www/html/(Path)/rra/asa_test_discards_in_549984.rrd"
rrd_version = "0003"
step = 300
last_update = 1667847988
header_size = 8240
ds[errors_in].index = 0
ds[errors_in].type = "COUNTER"
ds[errors_in].minimal_heartbeat = 120
ds[errors_in].min = 0.0000000000e+00
ds[errors_in].max = 1.0000000000e+07
ds[errors_in].last_ds = "1792801"
ds[errors_in].value = NaN
ds[errors_in].unknown_sec = 88
ds[discards_in].index = 1
ds[discards_in].type = "COUNTER"
ds[discards_in].minimal_heartbeat = 120
ds[discards_in].min = 0.0000000000e+00
ds[discards_in].max = 1.0000000000e+07
ds[discards_in].last_ds = "0"
ds[discards_in].value = NaN
ds[discards_in].unknown_sec = 88
ds[discards_out].index = 2
ds[discards_out].type = "COUNTER"
ds[discards_out].minimal_heartbeat = 120
ds[discards_out].min = 0.0000000000e+00
ds[discards_out].max = 1.0000000000e+07
ds[discards_out].last_ds = "0"
ds[discards_out].value = NaN
ds[discards_out].unknown_sec = 88
ds[errors_out].index = 3
ds[errors_out].type = "COUNTER"
ds[errors_out].minimal_heartbeat = 120
ds[errors_out].min = 0.0000000000e+00
ds[errors_out].max = 1.0000000000e+07
ds[errors_out].last_ds = "0"
ds[errors_out].value = NaN
ds[errors_out].unknown_sec = 88
rra[0].cf = "AVERAGE"
rra[0].rows = 600
rra[0].cur_row = 87
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[0].cdp_prep[2].value = NaN
rra[0].cdp_prep[2].unknown_datapoints = 0
rra[0].cdp_prep[3].value = NaN
rra[0].cdp_prep[3].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 6200
rra[1].cur_row = 5696
rra[1].pdp_per_row = 2
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = 0.0000000000e+00
rra[1].cdp_prep[0].unknown_datapoints = 1
rra[1].cdp_prep[1].value = 0.0000000000e+00
rra[1].cdp_prep[1].unknown_datapoints = 1
rra[1].cdp_prep[2].value = 0.0000000000e+00
rra[1].cdp_prep[2].unknown_datapoints = 1
rra[1].cdp_prep[3].value = 0.0000000000e+00
rra[1].cdp_prep[3].unknown_datapoints = 1
rra[2].cf = "AVERAGE"
rra[2].rows = 3250
rra[2].cur_row = 684
rra[2].pdp_per_row = 6
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = 0.0000000000e+00
rra[2].cdp_prep[0].unknown_datapoints = 1
rra[2].cdp_prep[1].value = 0.0000000000e+00
rra[2].cdp_prep[1].unknown_datapoints = 1
rra[2].cdp_prep[2].value = 0.0000000000e+00
rra[2].cdp_prep[2].unknown_datapoints = 1
rra[2].cdp_prep[3].value = 0.0000000000e+00
rra[2].cdp_prep[3].unknown_datapoints = 1
rra[3].cf = "AVERAGE"
rra[3].rows = 2242
rra[3].cur_row = 1150
rra[3].pdp_per_row = 48
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = 0.0000000000e+00
rra[3].cdp_prep[0].unknown_datapoints = 37
rra[3].cdp_prep[1].value = 0.0000000000e+00
rra[3].cdp_prep[1].unknown_datapoints = 37
rra[3].cdp_prep[2].value = 0.0000000000e+00
rra[3].cdp_prep[2].unknown_datapoints = 37
rra[3].cdp_prep[3].value = 0.0000000000e+00
rra[3].cdp_prep[3].unknown_datapoints = 37
rra[4].cf = "MIN"
rra[4].rows = 600
rra[4].cur_row = 439
rra[4].pdp_per_row = 1
rra[4].xff = 5.0000000000e-01
rra[4].cdp_prep[0].value = NaN
rra[4].cdp_prep[0].unknown_datapoints = 0
rra[4].cdp_prep[1].value = NaN
rra[4].cdp_prep[1].unknown_datapoints = 0
rra[4].cdp_prep[2].value = NaN
rra[4].cdp_prep[2].unknown_datapoints = 0
rra[4].cdp_prep[3].value = NaN
rra[4].cdp_prep[3].unknown_datapoints = 0
rra[5].cf = "MIN"
rra[5].rows = 6200
rra[5].cur_row = 1578
rra[5].pdp_per_row = 2
rra[5].xff = 5.0000000000e-01
rra[5].cdp_prep[0].value = inf
rra[5].cdp_prep[0].unknown_datapoints = 1
rra[5].cdp_prep[1].value = inf
rra[5].cdp_prep[1].unknown_datapoints = 1
rra[5].cdp_prep[2].value = inf
rra[5].cdp_prep[2].unknown_datapoints = 1
rra[5].cdp_prep[3].value = inf
rra[5].cdp_prep[3].unknown_datapoints = 1
rra[6].cf = "MIN"
rra[6].rows = 3250
rra[6].cur_row = 875
rra[6].pdp_per_row = 6
rra[6].xff = 5.0000000000e-01
rra[6].cdp_prep[0].value = inf
rra[6].cdp_prep[0].unknown_datapoints = 1
rra[6].cdp_prep[1].value = inf
rra[6].cdp_prep[1].unknown_datapoints = 1
rra[6].cdp_prep[2].value = inf
rra[6].cdp_prep[2].unknown_datapoints = 1
rra[6].cdp_prep[3].value = inf
rra[6].cdp_prep[3].unknown_datapoints = 1
rra[7].cf = "MIN"
rra[7].rows = 2242
rra[7].cur_row = 399
rra[7].pdp_per_row = 48
rra[7].xff = 5.0000000000e-01
rra[7].cdp_prep[0].value = inf
rra[7].cdp_prep[0].unknown_datapoints = 37
rra[7].cdp_prep[1].value = inf
rra[7].cdp_prep[1].unknown_datapoints = 37
rra[7].cdp_prep[2].value = inf
rra[7].cdp_prep[2].unknown_datapoints = 37
rra[7].cdp_prep[3].value = inf
rra[7].cdp_prep[3].unknown_datapoints = 37
rra[8].cf = "MAX"
rra[8].rows = 600
rra[8].cur_row = 550
rra[8].pdp_per_row = 1
rra[8].xff = 5.0000000000e-01
rra[8].cdp_prep[0].value = NaN
rra[8].cdp_prep[0].unknown_datapoints = 0
rra[8].cdp_prep[1].value = NaN
rra[8].cdp_prep[1].unknown_datapoints = 0
rra[8].cdp_prep[2].value = NaN
rra[8].cdp_prep[2].unknown_datapoints = 0
rra[8].cdp_prep[3].value = NaN
rra[8].cdp_prep[3].unknown_datapoints = 0
rra[9].cf = "MAX"
rra[9].rows = 6200
rra[9].cur_row = 977
rra[9].pdp_per_row = 2
rra[9].xff = 5.0000000000e-01
rra[9].cdp_prep[0].value = -inf
rra[9].cdp_prep[0].unknown_datapoints = 1
rra[9].cdp_prep[1].value = -inf
rra[9].cdp_prep[1].unknown_datapoints = 1
rra[9].cdp_prep[2].value = -inf
rra[9].cdp_prep[2].unknown_datapoints = 1
rra[9].cdp_prep[3].value = -inf
rra[9].cdp_prep[3].unknown_datapoints = 1
rra[10].cf = "MAX"
rra[10].rows = 3250
rra[10].cur_row = 2432
rra[10].pdp_per_row = 6
rra[10].xff = 5.0000000000e-01
rra[10].cdp_prep[0].value = -inf
rra[10].cdp_prep[0].unknown_datapoints = 1
rra[10].cdp_prep[1].value = -inf
rra[10].cdp_prep[1].unknown_datapoints = 1
rra[10].cdp_prep[2].value = -inf
rra[10].cdp_prep[2].unknown_datapoints = 1
rra[10].cdp_prep[3].value = -inf
rra[10].cdp_prep[3].unknown_datapoints = 1
rra[11].cf = "MAX"
rra[11].rows = 2242
rra[11].cur_row = 319
rra[11].pdp_per_row = 48
rra[11].xff = 5.0000000000e-01
rra[11].cdp_prep[0].value = -inf
rra[11].cdp_prep[0].unknown_datapoints = 37
rra[11].cdp_prep[1].value = -inf
rra[11].cdp_prep[1].unknown_datapoints = 37
rra[11].cdp_prep[2].value = -inf
rra[11].cdp_prep[2].unknown_datapoints = 37
rra[11].cdp_prep[3].value = -inf
rra[11].cdp_prep[3].unknown_datapoints = 37
rra[12].cf = "LAST"
rra[12].rows = 600
rra[12].cur_row = 181
rra[12].pdp_per_row = 1
rra[12].xff = 5.0000000000e-01
rra[12].cdp_prep[0].value = NaN
rra[12].cdp_prep[0].unknown_datapoints = 0
rra[12].cdp_prep[1].value = NaN
rra[12].cdp_prep[1].unknown_datapoints = 0
rra[12].cdp_prep[2].value = NaN
rra[12].cdp_prep[2].unknown_datapoints = 0
rra[12].cdp_prep[3].value = NaN
rra[12].cdp_prep[3].unknown_datapoints = 0
rra[13].cf = "LAST"
rra[13].rows = 6200
rra[13].cur_row = 4479
rra[13].pdp_per_row = 2
rra[13].xff = 5.0000000000e-01
rra[13].cdp_prep[0].value = NaN
rra[13].cdp_prep[0].unknown_datapoints = 1
rra[13].cdp_prep[1].value = NaN
rra[13].cdp_prep[1].unknown_datapoints = 1
rra[13].cdp_prep[2].value = NaN
rra[13].cdp_prep[2].unknown_datapoints = 1
rra[13].cdp_prep[3].value = NaN
rra[13].cdp_prep[3].unknown_datapoints = 1
rra[14].cf = "LAST"
rra[14].rows = 3250
rra[14].cur_row = 2723
rra[14].pdp_per_row = 6
rra[14].xff = 5.0000000000e-01
rra[14].cdp_prep[0].value = NaN
rra[14].cdp_prep[0].unknown_datapoints = 1
rra[14].cdp_prep[1].value = NaN
rra[14].cdp_prep[1].unknown_datapoints = 1
rra[14].cdp_prep[2].value = NaN
rra[14].cdp_prep[2].unknown_datapoints = 1
rra[14].cdp_prep[3].value = NaN
rra[14].cdp_prep[3].unknown_datapoints = 1
rra[15].cf = "LAST"
rra[15].rows = 2242
rra[15].cur_row = 826
rra[15].pdp_per_row = 48
rra[15].xff = 5.0000000000e-01
rra[15].cdp_prep[0].value = NaN
rra[15].cdp_prep[0].unknown_datapoints = 37
rra[15].cdp_prep[1].value = NaN
rra[15].cdp_prep[1].unknown_datapoints = 37
rra[15].cdp_prep[2].value = NaN
rra[15].cdp_prep[2].unknown_datapoints = 37
rra[15].cdp_prep[3].value = NaN
rra[15].cdp_prep[3].unknown_datapoints = 37

@TheWitness TheWitness added confirmed Bug is confirm by dev team and removed unverified Some days we don't have a clue labels Nov 8, 2022
@TheWitness TheWitness added this to the v1.2.23 milestone Nov 8, 2022
@TheWitness
Copy link
Member

Confirmed a few things with @bmfmancini tonight.

@TheWitness
Copy link
Member

Going to close this one. @bmfmancini will open two more bugs. We know the solution now. Just need to implement.

@TheWitness TheWitness reopened this Nov 8, 2022
@TheWitness
Copy link
Member

On second thought, going to rename the issue now that we know the source.

@TheWitness TheWitness changed the title [1.2.22] - Cisco ASA not plotting with 5 min interverals but 1 Min seems to be fine Updating a Data Template does not properly switch the rrd_heartbeat properly for all data sources resulting in empty graphs Nov 8, 2022
TheWitness added a commit that referenced this issue Nov 8, 2022
@TheWitness
Fixing #4991 - Data Source heartbeats
1c6e1e5 
Updating a Data Template does not properly switch the rrd_heartbeat properly for all data sources resulting in empty graphs
@TheWitness TheWitness added the resolved A fixed issue label Nov 8, 2022
TheWitness added a commit that referenced this issue Nov 8, 2022
@TheWitness
QA on #4991 - One additional inconsistency
24dc8af
@netniV netniV changed the title Updating a Data Template does not properly switch the rrd_heartbeat properly for all data sources resulting in empty graphs Updating a Data Template does not switch rrd_heartbeat properly for all sources leading to empty graphs Dec 31, 2022
@github-actions github-actions bot locked and limited conversation to collaborators Apr 1, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Undesired behaviour confirmed Bug is confirm by dev team resolved A fixed issue
Projects
None yet
Milestone
v1.2.23
Development

No branches or pull requests
2 participants
@TheWitness @bmfmancini