Allow Automatic Graph Creation to utilise Data Templates with Overriden Values

[eschoeller]

First, you have automatic graph creation turned on. Then, you create a new graph template which references a data template that has several values overridden ... say "Name", "Internal Data Source Name" and "OID".
Now you add this graph template to a host. Cacti immediately creates the graph. But it does not prompt you for the overridden values - so the Name, Internal Data Source Name and OID are all left blank

My work-around is to temporarily disable automatic graph creation, and manually create the graphs so all the data template prompts come up correctly.

But I'm not sure how to advise on this really ... perhaps there are cases where Cacti should prohibit the automatic creation of graphs? Or ... ?

[TheWitness]

Yea, I've been thinking about this for a while now. Your suggestion seems reasonable. Do your examples have defaults? I was thinking if the default is 'blank', that then we would not create automatically. Does that make sense?

[eschoeller]

My examples do not have defaults. So, when the data sources are created, they are very, very broken, and .. un-repairable from the GUI. I even get a database error splashed up on the data source edit screen.

So, likely the easiest avenue to pursue is to simply prohibit automatic creation if there are empty fields in the data source template. Would need to think about implications for empty fields in the graph template. It's probably just a few key fields that really are of the most concern - that would result in truly broken graphs/data sources.

I was thinking about a couple flip-side scenarios to this, but honestly they all seem to still result in creating data sources with problems. I think you really need to be actively creating the graphs to then be appropriately prompted to enter information to the empty fields on the data sources. The field entry is a manual process, so the graph creation should likely be manual too.

I would suggest that AUTOM8 throw a log entry indicating that it's skipping over certain graphs since they need manual user input. That will help with future questions of "Hey why didn't these graphs get created for me?"

I do think that if the fields are overridden, but they do have a default value (ie. not empty), then perhaps the intent of the template designer was to have the graphs created with those defaults. At least the data sources are created with valid information, and the user can then go back and tweak the values if they choose to.

[TheWitness]

Fixed.

[TheWitness]

Okay, tested on my side now and works.

[anarkia1976]

Okay, tested on my side now and works.

Fixed, thanks a lot.

[anarkia1976]

Hi @TheWitness , no error now but graphs aren't created after autom application:
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] automation_execute_device_create_tree() Device[TEST DEVICE], matching rule sql: SELECT atr.id, atr.name, atr.tree_id, atr.tree_item_id, atr.leaf_type, atr.host_grouping_type FROM automation_tree_rules AS atr WHERE enabled='on' AND leaf_type=3 matches: 0
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] automation_execute_device_create_tree() Device[TEST DEVICE] called
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] automation_update_device() Create Tree for Device[TEST DEVICE]
2021-03-26 10:56:22 - AUTOM8 NOTE: Device[TEST DEVICE], GT[Interface - Unicast], DQ[SNMP - Interface Statistics], Index[8], Rule[FORTINET CLUSTER - INTERFACE - UNICAST] Graph not added due to invalid data returned.
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] create_dq_graphs() Device[TEST DEVICE] - checking index: 8
2021-03-26 10:56:22 - AUTOM8 NOTE: Device[TEST DEVICE], GT[Interface - Unicast], DQ[SNMP - Interface Statistics], Index[3], Rule[FORTINET CLUSTER - INTERFACE - UNICAST] Graph not added due to invalid data returned.
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] create_dq_graphs() Device[TEST DEVICE] - checking index: 3
2021-03-26 10:56:22 - AUTOM8 NOTE: Device[TEST DEVICE], GT[Interface - Unicast], DQ[SNMP - Interface Statistics], Index[25], Rule[FORTINET CLUSTER - INTERFACE - UNICAST] Graph not added due to invalid data returned.
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] create_dq_graphs() Device[TEST DEVICE] - checking index: 25
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] create_dq_graphs() Found Template for Device[TEST DEVICE] - GT[Interface - Unicast]
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] build_rule_item_filter() returns: a.ifOperStatus <=> 'Up'
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] build_rule_item_filter() called: [{"id":"14","rule_id":"12","sequence":"1","operation":"0","field":"ifOperStatus","operator":"7","pattern":"Up"}], prefix: a.
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] create_dq_graphs() Device[TEST DEVICE] - snmp query: 1 - rule: FORTINET CLUSTER - INTERFACE - UNICAST
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] automation_execute_data_query() Device[TEST DEVICE] - create sql: SELECT h.id AS host_id, h.hostname, h.description, ht.name AS host_template_name FROM host AS h LEFT JOIN host_template AS ht ON h.host_template_id = ht.id WHERE ( h.description IS NOT NULL AND h.description LIKE '%CLUSTER%' OR h.description LIKE '%SINGLE%') AND h.id=1836 AND h.deleted = "" matches: 1
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] build_matching_objects_filter() returns: h.description IS NOT NULL AND h.description LIKE '%CLUSTER%' OR h.description LIKE '%SINGLE%'
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] build_rule_item_filter() returns: h.description IS NOT NULL AND h.description LIKE '%CLUSTER%' OR h.description LIKE '%SINGLE%'
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] build_rule_item_filter() called: [{"id":"21","rule_id":"12","rule_type":"1","sequence":"1","operation":"0","field":"h.description","operator":"14","pattern":""},{"id":"22","rule_id":"12","rule_type":"1","sequence":"2","operation":"1","field":"h.description","operator":"1","pattern":"CLUSTER"},{"id":"24","rule_id":"12","rule_type":"1","sequence":"3","operation":"2","field":"h.description","operator":"1","pattern":"SINGLE"}], prefix:
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] build_matching_objects_filter() called rule id: 12
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] automation_execute_data_query() Device[TEST DEVICE] - rule=12 name: FORTINET CLUSTER - INTERFACE - UNICAST
2021-03-26 10:56:22 - AUTOM8 NOTE: Device[TEST DEVICE], GT[Interface - Traffic (bits/sec)], DQ[SNMP - Interface Statistics], Index[8], Rule[FORTINET CLUSTER - INTERFACE - TRAFFIC] Graph not added due to invalid data returned.
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] create_dq_graphs() Device[TEST DEVICE] - checking index: 8
2021-03-26 10:56:22 - AUTOM8 NOTE: Device[TEST DEVICE], GT[Interface - Traffic (bits/sec)], DQ[SNMP - Interface Statistics], Index[3], Rule[FORTINET CLUSTER - INTERFACE - TRAFFIC] Graph not added due to invalid data returned.
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] create_dq_graphs() Device[TEST DEVICE] - checking index: 3
2021-03-26 10:56:22 - AUTOM8 NOTE: Device[TEST DEVICE], GT[Interface - Traffic (bits/sec)], DQ[SNMP - Interface Statistics], Index[25], Rule[FORTINET CLUSTER - INTERFACE - TRAFFIC] Graph not added due to invalid data returned.
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] create_dq_graphs() Device[TEST DEVICE] - checking index: 25
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] create_dq_graphs() Found Template for Device[TEST DEVICE] - GT[Interface - Traffic (bits/sec)]
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] build_rule_item_filter() returns: a.ifOperStatus <=> 'Up'
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] build_rule_item_filter() called: [{"id":"13","rule_id":"11","sequence":"1","operation":"0","field":"ifOperStatus","operator":"7","pattern":"Up"}], prefix: a.
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] create_dq_graphs() Device[TEST DEVICE] - snmp query: 1 - rule: FORTINET CLUSTER - INTERFACE - TRAFFIC
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] automation_execute_data_query() Device[TEST DEVICE] - create sql: SELECT h.id AS host_id, h.hostname, h.description, ht.name AS host_template_name FROM host AS h LEFT JOIN host_template AS ht ON h.host_template_id = ht.id WHERE ( h.description IS NOT NULL AND h.description LIKE '%CLUSTER%' OR h.description LIKE '%SINGLE%') AND h.id=1836 AND h.deleted = "" matches: 1
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] build_matching_objects_filter() returns: h.description IS NOT NULL AND h.description LIKE '%CLUSTER%' OR h.description LIKE '%SINGLE%'
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] build_rule_item_filter() returns: h.description IS NOT NULL AND h.description LIKE '%CLUSTER%' OR h.description LIKE '%SINGLE%'
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] build_rule_item_filter() called: [{"id":"19","rule_id":"11","rule_type":"1","sequence":"1","operation":"0","field":"h.description","operator":"14","pattern":""},{"id":"20","rule_id":"11","rule_type":"1","sequence":"2","operation":"1","field":"h.description","operator":"1","pattern":"CLUSTER"},{"id":"23","rule_id":"11","rule_type":"1","sequence":"3","operation":"2","field":"h.description","operator":"1","pattern":"SINGLE"}], prefix:
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] build_matching_objects_filter() called rule id: 11
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] automation_execute_data_query() Device[TEST DEVICE] - rule=11 name: FORTINET CLUSTER - INTERFACE - TRAFFIC
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] automation_execute_data_query() Device[TEST DEVICE] - sql: SELECT agr.id, agr.name, agr.snmp_query_id, agr.graph_type_id FROM automation_graph_rules AS agr INNER JOIN host_snmp_query AS hsq ON agr.snmp_query_id = hsq.snmp_query_id WHERE agr.snmp_query_id = ? AND hsq.host_id = ? AND enabled="on" - found: 2
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] automation_execute_data_query() Device[TEST DEVICE] - start - data query: 1
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] automation_update_device() Found DQ[SNMP - Interface Statistics] for Device[TEST DEVICE]
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] automation_update_device() Device[TEST DEVICE], sql: SELECT gt.* FROM graph_templates AS gt INNER JOIN host_graph AS hg ON gt.id=hg.graph_template_id WHERE hg.host_id=1836 AND gt.id NOT IN ( SELECT gl.graph_template_id FROM graph_local AS gl WHERE host_id=1836 )
2021-03-26 10:56:22 - AUTOM8 TRACE [PID: 1746671] automation_update_device() Device[TEST DEVICE]
2021-03-26 10:56:22 - AUTOM8 TRACE form_actions, items: a:1:{i:0;s:4:"1836";}
2021-03-26 10:56:22 - AUTOM8 TRACE form_actions called, action: 6

Graph rules are matched correctly. Thanks a lot.

[TheWitness]

Okay, should be working now.

[TheWitness]

Okay, I have tested all methods and creation is blocked not only during automation, but manual graph creation.

[TheWitness]

Actually, just found an issue.

[TheWitness]

Testing completed now.

[anarkia1976]

It's ok now, thanks a lot.