Updating a Data Template does not switch rrd_heartbeat properly for all sources leading to empty graphs

[bmfmancini]

Hey Guys,

I am seeing a weird issue with the following

Cisco ASA using the ASA template
SNMP is showing up and spine collects data but nothing plots on any snmp graph
Only graphs that work as the snmp uptime, polling time and ping
I have the same device on the same cacti version but polling every minute working without issue

Here is the Spine Data

2022-11-07 12:09:21 - SPINE: Poller[1] PID[619685] PT[140299301230336] Device[17147] HT[1] DS[549984] TT[14.80] SNMP: v3: (device_ip), dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.10, value: 1792801
2022-11-07 12:09:21 - SPINE: Poller[1] PID[619685] PT[140299301230336] Device[17147] HT[1] DS[549984] TT[14.81] SNMP: v3: (device_ip), dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.10, value: 0
2022-11-07 12:09:21 - SPINE: Poller[1] PID[619685] PT[140299301230336] Device[17147] HT[1] DS[549984] TT[14.82] SNMP: v3: (device_ip), dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.10, value: 0
2022-11-07 12:09:21 - SPINE: Poller[1] PID[619685] PT[140299301230336] Device[17147] HT[1] DS[549984] TT[14.83] SNMP: v3: (device_ip), dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.10, value: 0

Here is the Data source debug output shows that value is NaN

the data template is set to 5Min polling as it should be

Poller cache is showing the proper OID's

ASA test - Errors - GigabitEthernet0/7	ASA test	SNMP Version: 3, User: (snmp_user), OID: .1.3.6.1.2.1.2.2.1.13.10
RRD: /var/www/html/(path)/rra/asa_test_discards_in_549984.rrd
ASA test - Errors - GigabitEthernet0/7	ASA test	SNMP Version: 3, User: (snmp_user), OID: .1.3.6.1.2.1.2.2.1.19.10
RRD: /var/www/html/(path)/rra/asa_test_discards_in_549984.rrd
ASA test - Errors - GigabitEthernet0/7	ASA test	SNMP Version: 3, User: (snmp_user), OID: .1.3.6.1.2.1.2.2.1.14.10
RRD: /var/www/html/(path)/rra/asa_test_discards_in_549984.rrd
ASA test - Errors - GigabitEthernet0/7	ASA test	SNMP Version: 3, User: (snmp_user), OID: .1.3.6.1.2.1.2.2.1.20.10
RRD: /var/www/html/(path)/rra/asa_test_discards_in_549984.rrd

PHP-snmp is not installed

RRD permission's are correct

-rw-r--r--  1 apache apache 388K Nov  7 11:58 asa_test_5min_cpu_549976.rrd
-rw-r--r--  1 apache apache 1.6M Nov  7 11:58 asa_test_discards_in_549984.rrd
-rw-r--r--  1 apache apache 1.6M Nov  7 11:58 asa_test_discards_in_549985.rrd
-rw-r--r--  1 apache apache 1.6M Nov  7 11:58 asa_test_discards_in_549986.rrd
-rw-r--r--  1 apache apache 1.6M Nov  7 11:58 asa_test_discards_in_549987.rrd
-rw-r--r--  1 apache apache 1.6M Nov  7 11:58 asa_test_discards_in_549988.rrd
-rw-r--r--  1 apache apache 388K Nov  7 11:58 asa_test_polling_time_549977.rrd
-rw-r--r--  1 apache apache 774K Nov  7 11:58 asa_test_traffic_in_549979.rrd
-rw-r--r--  1 apache apache 774K Nov  7 11:58 asa_test_traffic_in_549980.rrd
-rw-r--r--  1 apache apache 774K Nov  7 11:58 asa_test_traffic_in_549981.rrd
-rw-r--r--  1 apache apache 774K Nov  7 11:58 asa_test_traffic_in_549982.rrd
-rw-r--r--  1 apache apache 774K Nov  7 11:58 asa_test_traffic_in_549983.rrd
-rw-r--r--  1 apache apache 388K Nov  7 11:58 asa_test_uptime_549978.rrd

RRD's are set to the correct step

filename = "asa_test_discards_in_549984.rrd"
rrd_version = "0003"
step = 300
last_update = 1667839887
header_size = 8240

Boost has the collected results waiting to be written

MariaDB [cacti]> select * from poller_output_boost where local_data_id = 549984;
+---------------+--------------+---------------------+---------+
| local_data_id | rrd_name     | time                | output  |
+---------------+--------------+---------------------+---------+
|        549984 | discards_in  | 2022-11-07 11:56:26 | 0       |
|        549984 | discards_out | 2022-11-07 11:56:26 | 0       |
|        549984 | errors_in    | 2022-11-07 11:56:26 | 1792801 |
|        549984 | errors_out   | 2022-11-07 11:56:26 | 0       |
|        549984 | discards_in  | 2022-11-07 12:01:29 | 0       |
|        549984 | discards_out | 2022-11-07 12:01:29 | 0       |
|        549984 | errors_in    | 2022-11-07 12:01:29 | 1792801 |
|        549984 | errors_out   | 2022-11-07 12:01:29 | 0       |
|        549984 | discards_in  | 2022-11-07 12:06:29 | 0       |
|        549984 | discards_out | 2022-11-07 12:06:29 | 0       |
|        549984 | errors_in    | 2022-11-07 12:06:29 | 1792801 |

[bmfmancini]

the troubleshooter seems to think everything is fine

[bmfmancini]

Tried another ASA with snmpv2 no difference

[TheWitness]

Post the RRDtool info for one of the RRDfiles.

[bmfmancini]

84.rrd
filename = "/var/www/html/(Path)/rra/asa_test_discards_in_549984.rrd"
rrd_version = "0003"
step = 300
last_update = 1667847988
header_size = 8240
ds[errors_in].index = 0
ds[errors_in].type = "COUNTER"
ds[errors_in].minimal_heartbeat = 120
ds[errors_in].min = 0.0000000000e+00
ds[errors_in].max = 1.0000000000e+07
ds[errors_in].last_ds = "1792801"
ds[errors_in].value = NaN
ds[errors_in].unknown_sec = 88
ds[discards_in].index = 1
ds[discards_in].type = "COUNTER"
ds[discards_in].minimal_heartbeat = 120
ds[discards_in].min = 0.0000000000e+00
ds[discards_in].max = 1.0000000000e+07
ds[discards_in].last_ds = "0"
ds[discards_in].value = NaN
ds[discards_in].unknown_sec = 88
ds[discards_out].index = 2
ds[discards_out].type = "COUNTER"
ds[discards_out].minimal_heartbeat = 120
ds[discards_out].min = 0.0000000000e+00
ds[discards_out].max = 1.0000000000e+07
ds[discards_out].last_ds = "0"
ds[discards_out].value = NaN
ds[discards_out].unknown_sec = 88
ds[errors_out].index = 3
ds[errors_out].type = "COUNTER"
ds[errors_out].minimal_heartbeat = 120
ds[errors_out].min = 0.0000000000e+00
ds[errors_out].max = 1.0000000000e+07
ds[errors_out].last_ds = "0"
ds[errors_out].value = NaN
ds[errors_out].unknown_sec = 88
rra[0].cf = "AVERAGE"
rra[0].rows = 600
rra[0].cur_row = 87
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[0].cdp_prep[2].value = NaN
rra[0].cdp_prep[2].unknown_datapoints = 0
rra[0].cdp_prep[3].value = NaN
rra[0].cdp_prep[3].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 6200
rra[1].cur_row = 5696
rra[1].pdp_per_row = 2
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = 0.0000000000e+00
rra[1].cdp_prep[0].unknown_datapoints = 1
rra[1].cdp_prep[1].value = 0.0000000000e+00
rra[1].cdp_prep[1].unknown_datapoints = 1
rra[1].cdp_prep[2].value = 0.0000000000e+00
rra[1].cdp_prep[2].unknown_datapoints = 1
rra[1].cdp_prep[3].value = 0.0000000000e+00
rra[1].cdp_prep[3].unknown_datapoints = 1
rra[2].cf = "AVERAGE"
rra[2].rows = 3250
rra[2].cur_row = 684
rra[2].pdp_per_row = 6
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = 0.0000000000e+00
rra[2].cdp_prep[0].unknown_datapoints = 1
rra[2].cdp_prep[1].value = 0.0000000000e+00
rra[2].cdp_prep[1].unknown_datapoints = 1
rra[2].cdp_prep[2].value = 0.0000000000e+00
rra[2].cdp_prep[2].unknown_datapoints = 1
rra[2].cdp_prep[3].value = 0.0000000000e+00
rra[2].cdp_prep[3].unknown_datapoints = 1
rra[3].cf = "AVERAGE"
rra[3].rows = 2242
rra[3].cur_row = 1150
rra[3].pdp_per_row = 48
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = 0.0000000000e+00
rra[3].cdp_prep[0].unknown_datapoints = 37
rra[3].cdp_prep[1].value = 0.0000000000e+00
rra[3].cdp_prep[1].unknown_datapoints = 37
rra[3].cdp_prep[2].value = 0.0000000000e+00
rra[3].cdp_prep[2].unknown_datapoints = 37
rra[3].cdp_prep[3].value = 0.0000000000e+00
rra[3].cdp_prep[3].unknown_datapoints = 37
rra[4].cf = "MIN"
rra[4].rows = 600
rra[4].cur_row = 439
rra[4].pdp_per_row = 1
rra[4].xff = 5.0000000000e-01
rra[4].cdp_prep[0].value = NaN
rra[4].cdp_prep[0].unknown_datapoints = 0
rra[4].cdp_prep[1].value = NaN
rra[4].cdp_prep[1].unknown_datapoints = 0
rra[4].cdp_prep[2].value = NaN
rra[4].cdp_prep[2].unknown_datapoints = 0
rra[4].cdp_prep[3].value = NaN
rra[4].cdp_prep[3].unknown_datapoints = 0
rra[5].cf = "MIN"
rra[5].rows = 6200
rra[5].cur_row = 1578
rra[5].pdp_per_row = 2
rra[5].xff = 5.0000000000e-01
rra[5].cdp_prep[0].value = inf
rra[5].cdp_prep[0].unknown_datapoints = 1
rra[5].cdp_prep[1].value = inf
rra[5].cdp_prep[1].unknown_datapoints = 1
rra[5].cdp_prep[2].value = inf
rra[5].cdp_prep[2].unknown_datapoints = 1
rra[5].cdp_prep[3].value = inf
rra[5].cdp_prep[3].unknown_datapoints = 1
rra[6].cf = "MIN"
rra[6].rows = 3250
rra[6].cur_row = 875
rra[6].pdp_per_row = 6
rra[6].xff = 5.0000000000e-01
rra[6].cdp_prep[0].value = inf
rra[6].cdp_prep[0].unknown_datapoints = 1
rra[6].cdp_prep[1].value = inf
rra[6].cdp_prep[1].unknown_datapoints = 1
rra[6].cdp_prep[2].value = inf
rra[6].cdp_prep[2].unknown_datapoints = 1
rra[6].cdp_prep[3].value = inf
rra[6].cdp_prep[3].unknown_datapoints = 1
rra[7].cf = "MIN"
rra[7].rows = 2242
rra[7].cur_row = 399
rra[7].pdp_per_row = 48
rra[7].xff = 5.0000000000e-01
rra[7].cdp_prep[0].value = inf
rra[7].cdp_prep[0].unknown_datapoints = 37
rra[7].cdp_prep[1].value = inf
rra[7].cdp_prep[1].unknown_datapoints = 37
rra[7].cdp_prep[2].value = inf
rra[7].cdp_prep[2].unknown_datapoints = 37
rra[7].cdp_prep[3].value = inf
rra[7].cdp_prep[3].unknown_datapoints = 37
rra[8].cf = "MAX"
rra[8].rows = 600
rra[8].cur_row = 550
rra[8].pdp_per_row = 1
rra[8].xff = 5.0000000000e-01
rra[8].cdp_prep[0].value = NaN
rra[8].cdp_prep[0].unknown_datapoints = 0
rra[8].cdp_prep[1].value = NaN
rra[8].cdp_prep[1].unknown_datapoints = 0
rra[8].cdp_prep[2].value = NaN
rra[8].cdp_prep[2].unknown_datapoints = 0
rra[8].cdp_prep[3].value = NaN
rra[8].cdp_prep[3].unknown_datapoints = 0
rra[9].cf = "MAX"
rra[9].rows = 6200
rra[9].cur_row = 977
rra[9].pdp_per_row = 2
rra[9].xff = 5.0000000000e-01
rra[9].cdp_prep[0].value = -inf
rra[9].cdp_prep[0].unknown_datapoints = 1
rra[9].cdp_prep[1].value = -inf
rra[9].cdp_prep[1].unknown_datapoints = 1
rra[9].cdp_prep[2].value = -inf
rra[9].cdp_prep[2].unknown_datapoints = 1
rra[9].cdp_prep[3].value = -inf
rra[9].cdp_prep[3].unknown_datapoints = 1
rra[10].cf = "MAX"
rra[10].rows = 3250
rra[10].cur_row = 2432
rra[10].pdp_per_row = 6
rra[10].xff = 5.0000000000e-01
rra[10].cdp_prep[0].value = -inf
rra[10].cdp_prep[0].unknown_datapoints = 1
rra[10].cdp_prep[1].value = -inf
rra[10].cdp_prep[1].unknown_datapoints = 1
rra[10].cdp_prep[2].value = -inf
rra[10].cdp_prep[2].unknown_datapoints = 1
rra[10].cdp_prep[3].value = -inf
rra[10].cdp_prep[3].unknown_datapoints = 1
rra[11].cf = "MAX"
rra[11].rows = 2242
rra[11].cur_row = 319
rra[11].pdp_per_row = 48
rra[11].xff = 5.0000000000e-01
rra[11].cdp_prep[0].value = -inf
rra[11].cdp_prep[0].unknown_datapoints = 37
rra[11].cdp_prep[1].value = -inf
rra[11].cdp_prep[1].unknown_datapoints = 37
rra[11].cdp_prep[2].value = -inf
rra[11].cdp_prep[2].unknown_datapoints = 37
rra[11].cdp_prep[3].value = -inf
rra[11].cdp_prep[3].unknown_datapoints = 37
rra[12].cf = "LAST"
rra[12].rows = 600
rra[12].cur_row = 181
rra[12].pdp_per_row = 1
rra[12].xff = 5.0000000000e-01
rra[12].cdp_prep[0].value = NaN
rra[12].cdp_prep[0].unknown_datapoints = 0
rra[12].cdp_prep[1].value = NaN
rra[12].cdp_prep[1].unknown_datapoints = 0
rra[12].cdp_prep[2].value = NaN
rra[12].cdp_prep[2].unknown_datapoints = 0
rra[12].cdp_prep[3].value = NaN
rra[12].cdp_prep[3].unknown_datapoints = 0
rra[13].cf = "LAST"
rra[13].rows = 6200
rra[13].cur_row = 4479
rra[13].pdp_per_row = 2
rra[13].xff = 5.0000000000e-01
rra[13].cdp_prep[0].value = NaN
rra[13].cdp_prep[0].unknown_datapoints = 1
rra[13].cdp_prep[1].value = NaN
rra[13].cdp_prep[1].unknown_datapoints = 1
rra[13].cdp_prep[2].value = NaN
rra[13].cdp_prep[2].unknown_datapoints = 1
rra[13].cdp_prep[3].value = NaN
rra[13].cdp_prep[3].unknown_datapoints = 1
rra[14].cf = "LAST"
rra[14].rows = 3250
rra[14].cur_row = 2723
rra[14].pdp_per_row = 6
rra[14].xff = 5.0000000000e-01
rra[14].cdp_prep[0].value = NaN
rra[14].cdp_prep[0].unknown_datapoints = 1
rra[14].cdp_prep[1].value = NaN
rra[14].cdp_prep[1].unknown_datapoints = 1
rra[14].cdp_prep[2].value = NaN
rra[14].cdp_prep[2].unknown_datapoints = 1
rra[14].cdp_prep[3].value = NaN
rra[14].cdp_prep[3].unknown_datapoints = 1
rra[15].cf = "LAST"
rra[15].rows = 2242
rra[15].cur_row = 826
rra[15].pdp_per_row = 48
rra[15].xff = 5.0000000000e-01
rra[15].cdp_prep[0].value = NaN
rra[15].cdp_prep[0].unknown_datapoints = 37
rra[15].cdp_prep[1].value = NaN
rra[15].cdp_prep[1].unknown_datapoints = 37
rra[15].cdp_prep[2].value = NaN
rra[15].cdp_prep[2].unknown_datapoints = 37
rra[15].cdp_prep[3].value = NaN
rra[15].cdp_prep[3].unknown_datapoints = 37

[TheWitness]

Confirmed a few things with @bmfmancini tonight.

[TheWitness]

Going to close this one. @bmfmancini will open two more bugs. We know the solution now. Just need to implement.

[TheWitness]

On second thought, going to rename the issue now that we know the source.