Unable to create graphs due to Data Source verification failure

[ssvenn]

I ran into the problem with "Damaged Graph" on 1.2.16 after editing my new templates for Brocade FC switches, and tried to fix it by upgrading to 1.2.17 - this didn't help so i erased the rrd files, reset the database and started from scratch with my exported template

I am seeing strange behaviour where only some of my queried interfaces are being created, most fail with the text
"NOTE: Graph not for Data Query and index due to Data Source verification failure.

Rolling back to 1.2.16 and resetting the database again I was able to use the template to create graphs without getting this error message, and spine is able to collect values.

cacti_data_query_brocade_fcmgmt_switch.zip

Steps to reproduce the behavior:

1. Import the included templates

2. Create a Brocade FC Switch device

3. Create graphs from the the Brocade FCMGMT Switch query

[ssvenn]

This might be related to feature#645: Modify automation to test for data before creating graphs

the 64-bit traffic counter on Brocade switches respond with Hex-String data on these OIDs that only work correctly with Spine, the PHP poller seems unable to convert it to numerical values.

snmpwalk -m FCMGMT-MIB -v2c -cpublic 192.168.1.10 .1.3.6.1.3.94.4.5.1.7 -O fn
.1.3.6.1.3.94.4.5.1.7.16.0.0.5.51.253.167.72.0.0.0.0.0.0.0.0.1 = Hex-STRING: 00 00 03 AB A0 BA 1D CC
.1.3.6.1.3.94.4.5.1.7.16.0.0.5.51.253.167.72.0.0.0.0.0.0.0.0.2 = Hex-STRING: 00 00 05 4A EE 5E 3B 24
.1.3.6.1.3.94.4.5.1.7.16.0.0.5.51.253.167.72.0.0.0.0.0.0.0.0.3 = Hex-STRING: 00 00 02 0D 66 93 01 68

https://forums.cacti.net/viewtopic.php?f=21&t=61663 has some more background

[Lejooohn]

Hi,

After upgrade from 1.2.16 to 1.2.17 i have a similar problem with ucd/net template :

Someone have a solution to fix that or we have to wait a next update?

Regards,

[xmacan]

@Lejooohn please show result of snmpwalk:
snmpwalk -c YOUR_COMMUNITY HOST_IP_ADDRESS 1.3.6.1.4.1.2021.4

[Lejooohn]

@Lejooohn please show result of snmpwalk:
snmpwalk -c YOUR_COMMUNITY HOST_IP_ADDRESS 1.3.6.1.4.1.2021.4

Hello @xmacan ,

Below, the output :

UCD-SNMP-MIB::memIndex.0 = INTEGER: 0
UCD-SNMP-MIB::memErrorName.0 = STRING: swap
UCD-SNMP-MIB::memTotalSwap.0 = INTEGER: 2047996 kB
UCD-SNMP-MIB::memAvailSwap.0 = INTEGER: 2047996 kB
UCD-SNMP-MIB::memTotalReal.0 = INTEGER: 4041112 kB
UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 3264592 kB
UCD-SNMP-MIB::memTotalFree.0 = INTEGER: 5312588 kB
UCD-SNMP-MIB::memMinimumSwap.0 = INTEGER: 16000 kB
UCD-SNMP-MIB::memShared.0 = INTEGER: 5524 kB
UCD-SNMP-MIB::memBuffer.0 = INTEGER: 2392 kB
UCD-SNMP-MIB::memCached.0 = INTEGER: 549568 kB
UCD-SNMP-MIB::memSwapError.0 = INTEGER: noError(0)
UCD-SNMP-MIB::memSwapErrorMsg.0 = STRING:

Regards,

John

[ssvenn]

I think a refinement of this data validation feature would be to stop the automatic countdown on that dialog box so that users have more time to act on it, and add a "force create graphs" checkbox somewhere as a workaround for edge cases like mine where Spine polls valid data while PHP does not.

[netniV]

The idea behind the automated countdown is to prevent you from needing to press anything to continue, but in this case it does sound like it needs to be a manual process. Can you paste a screenshot of the exact dialog you had, if it isn't this one?

[ssvenn]

Yes it is that dialog box, I just had to cut off the bottom to hide some host names. My environment is currently rolled back to 1.2.16 but I can set up a new instance on 1.2.17 to test more if you need more details.

Validating that a data source produces usable values is definitely a good idea to help users avoid empty graphs and poller errors, but when a graph fails to create it would be good to get a clear and detailed cause. Perhaps just send the text from this dialog box to the log file and some more debug information including the actual polled value and what's wrong with it (empty value, snmp error, invalid data type etc)

[netniV]

Yeah, I think I am going to rope @TheWitness in here to give a comment or two just because I'm not 100% clear yet on how that is working and he'll be able to answer it better than me. Between us, we should be able to come up with a way to do that.

[simonpunk]

@netniV
That's possibly the same issue as mind: #4246, still no comments tho...
Anyway, I think the root cause is the defined template custom data field is used for the mechanism rather than the newly defined custom data field when user trying to create a graph. In Chrome debug session, I see it sends correct post value with my input after clicking the create graph button, but seems the php code looks up the databases input instead.

@ssvenn @Lejooohn I have tried to modify or insert a pre-defined value for my custom data field in databases, and it works, as well as creating another data source template with a pre-defined value for all the data field.

So the scene is like: even you are giving different value for your custom data fields, when you click create button, it doesn't take your input into account but instead the pre-defined input from your data source template. So only when the output returns no 'U' from your data source template will let you keep going.

[zuka1337]

Same here, 2 diferent types of error when creating any custom graph

First:

<html>
<body>
<!--StartFragment-->

2021-05-13 16:36:53 - CMDPHP PHP ERROR NOTICE Backtrace: (/graphs_new.php[42]:form_save(), /graphs_new.php[150]:html_graph_new_graphs(), /lib/html_graph.php[442]:html_graph_custom_data(), /lib/html_graph.php[573]:draw_nontemplated_fields_graph_item(), /lib/html_form_template.php[252]:draw_edit_form(), /lib/html_form.php[114]:draw_edit_control(), /lib/html_form.php[330]:CactiErrorHandler())
--
2021-05-13 16:36:53 - ERROR PHP NOTICE: Undefined index: value in file: /cacti/lib/html_form.php on line: 330
2021-05-13 16:36:53 - CMDPHP PHP ERROR NOTICE Backtrace: (/graphs_new.php[42]:form_save(), /graphs_new.php[150]:html_graph_new_graphs(), /lib/html_graph.php[442]:html_graph_custom_data(), /lib/html_graph.php[573]:draw_nontemplated_fields_graph_item(), /lib/html_form_template.php[181]:CactiErrorHandler())
2021-05-13 16:36:53 - ERROR PHP NOTICE: Undefined index: color_id in file: /cacti/lib/html_form_template.php on line: 181
2021-05-13 16:36:53 - CMDPHP PHP ERROR NOTICE Backtrace: (/graphs_new.php[42]:form_save(), /graphs_new.php[150]:html_graph_new_graphs(), /lib/html_graph.php[442]:html_graph_custom_data(), /lib/html_graph.php[573]:draw_nontemplated_fields_graph_item(), /lib/html_form_template.php[180]:CactiErrorHandler())
2021-05-13 16:36:53 - ERROR PHP NOTICE: Undefined index: color_id in file: /cacti/lib/html_form_template.php on line: 180

<!--EndFragment-->
</body>
</html>

Second:

[zuka1337]

And becomes impossible to add devices by cli with an associated template Device containning graphs that cannot be created by this error:

/usr/bin/php -q /cacti/cli/add_device.php --description=HOSTNAME--ip=IP --community=CM --template=X--version=2 --port=161 --avail=pingsnmp --disable=1

USAGE: snmpget [OPTIONS] AGENT OID [OID]...

  Version:  5.8
  Web:      http://www.net-snmp.org/
  Email:    net-snmp-coders@lists.sourceforge.net

OPTIONS:
  -h, --help            display this help message
  -H                    display configuration file directives understood
  -v 1|2c|3             specifies SNMP version to use
  -V, --version         display package version number
SNMP Version 1 or 2c specific
  -c COMMUNITY          set the community string
SNMP Version 3 specific
  -a PROTOCOL           set authentication protocol (MD5|SHA|SHA-224|SHA-256|SHA-384|SHA-512)
  -A PASSPHRASE         set authentication protocol pass phrase
  -e ENGINE-ID          set security engine ID (e.g. 800000020109840301)
  -E ENGINE-ID          set context engine ID (e.g. 800000020109840301)
  -l LEVEL              set security level (noAuthNoPriv|authNoPriv|authPriv)
  -n CONTEXT            set context name (e.g. bridge1)
  -u USER-NAME          set security name (e.g. bert)
  -x PROTOCOL           set privacy protocol (DES|AES|AES-192|AES-256)
  -X PASSPHRASE         set privacy protocol pass phrase
  -Z BOOTS,TIME         set destination engine boots/time
General communication options
  -r RETRIES            set the number of retries
  -t TIMEOUT            set the request timeout (in seconds)
Debugging
  -d                    dump input/output packets in hexadecimal
  -D[TOKEN[,...]]       turn on debugging output for the specified TOKENs
                           (ALL gives extremely verbose debugging output)
General options
  -m MIB[:...]          load given list of MIBs (ALL loads everything)
  -M DIR[:...]          look in given list of directories for MIBs
    (default: /root/.snmp/mibs:/usr/share/snmp/mibs)
  -P MIBOPTS            Toggle various defaults controlling MIB parsing:
                          u:  allow the use of underlines in MIB symbols
                          c:  disallow the use of "--" to terminate comments
                          d:  save the DESCRIPTIONs of the MIB objects
                          e:  disable errors when MIB symbols conflict
                          w:  enable warnings when MIB symbols conflict
                          W:  enable detailed warnings when MIB symbols conflict
                          R:  replace MIB symbols from latest module
  -O OUTOPTS            Toggle various defaults controlling output display:
                          0:  print leading 0 for single-digit hex characters
                          a:  print all strings in ascii format
                          b:  do not break OID indexes down
                          e:  print enums numerically
                          E:  escape quotes in string indices
                          f:  print full OIDs on output
                          n:  print OIDs numerically
                          p PRECISION:  display floating point values with specified PRECISION (printf format string)
                          q:  quick print for easier parsing
                          Q:  quick print with equal-signs
                          s:  print only last symbolic element of OID
                          S:  print MIB module-id plus last element
                          t:  print timeticks unparsed as numeric integers
                          T:  print human-readable text along with hex strings
                          u:  print OIDs using UCD-style prefix suppression
                          U:  don't print units
                          v:  print values only (not OID = value)
                          x:  print all strings in hex format
                          X:  extended index format
  -I INOPTS             Toggle various defaults controlling input parsing:
                          b:  do best/regex matching to find a MIB node
                          h:  don't apply DISPLAY-HINTs
                          r:  do not check values for range/type legality
                          R:  do random access to OID labels
                          u:  top-level OIDs must have '.' prefix (UCD-style)
                          s SUFFIX:  Append all textual OIDs with SUFFIX before parsing
                          S PREFIX:  Prepend all textual OIDs with PREFIX before parsing
  -L LOGOPTS            Toggle various defaults controlling logging:
                          e:           log to standard error
                          o:           log to standard output
                          n:           don't log at all
                          f file:      log to the specified file
                          s facility:  log to syslog (via the specified facility)

                          (variants)
                          [EON] pri:   log to standard error, output or /dev/null for level 'pri' and above
                          [EON] p1-p2: log to standard error, output or /dev/null for levels 'p1' to 'p2'
                          [FS] pri token:    log to file/syslog for level 'pri' and above
                          [FS] p1-p2 token:  log to file/syslog for levels 'p1' to 'p2'
  -C APPOPTS            Set various application specific behaviours:
                          f:  do not fix errors and retry the request
No hostname specified.

At the end you're still gonna ending up with the device added, but the erros will mess your eyes.
The input scripts runs fine manually

Note: Nothing is bad with snmp (I know you're thinking :D)

[xmacan]

@zuka1337 I have similar problem problem Try this: console -> Templates -> Graph -> Edit one of your problematic graphs (Fan or memory heap) -> tick any checkbox (I tried upper limit, it does not matter). Try to create graphs again and let me know.

[TheWitness]

Sorry guys, been taking a break. Needed some time off. Tried getting this remediated over the weekend, but my computer took one look at me and ran for the hills.