W.I.P. PR2: New functions created to query for each item - calling of functions in importer.pp #10

liamjohnsexton · 2020-12-08T11:12:13Z

Hey guys,

In this pull request:

Added function to query for puppet servers on the estate
Added function to query for puppetdb_hosts on the estate
Added function to query for postgres_hosts on the estate (this one I'm not too confident on, essentially copied the code for querying the puppetdb_hosts, but changed to look for nodes with the class pe_postgresql::server::install attached)
Changed the code in importer.pp to call each of these functions and save them to their respective variable names, then calling these variables in the puppet_metrics_dashboard class

dylanratcliffe

Looks great. Only thing I would change is making the variables parameters so that it's easy to customise

dylanratcliffe · 2020-12-09T13:09:25Z

manifests/importer.pp

+
+  #in the below class 
+
+    $puppet_severs = rsan::get_puppet_servers()


Can we expose these as parameters instead of just declaring as variables? That way someone should be able to call the class like so and override the defaults if they need to:

class { 'rsan::importer': puppet_servers => ['server1.company.com', 'server2.company.com'], }

So in this instance, I'd be looking to do something like:

class { 'rsan::importer': puppet_servers => $puppet_servers, puppetdb_hosts => $puppetdb_hosts, postgres_hosts => $postgres_hosts, }

Is that right?

Well that's what the user would do when the use RSAN. In order for you to allow them to do that you'll need to allow the rsan::importer class to accept parameters that the user passes i.e.

class rsan::importer ( String $some_param = "hello", ) {

MartyEwings · 2020-12-09T13:51:23Z

functions/get_postgres_hosts.pp

@@ -0,0 +1,18 @@
+function rsan::get_postgres_hosts() {
+  if $settings::storeconfigs {


this makes this whole thing conditional on $settings::storeconfigs, which is a snippet from another place in code this was copied from. This does not need to be conditional

MartyEwings · 2020-12-09T13:52:08Z

functions/get_postgres_hosts.pp

+                      expired is null
+                    }
+                  }').map |$data| { $data['certname'] }
+  } else {


as there is no longer a conditional the exception of not having a postgres host returned needs handled differently

Hey Marty, would this mean I need to implement some sort of error catch? So if puppetdb_query returns nothing/an error - make an exception to set

$postgres_hosts = []

MartyEwings · 2020-12-09T13:52:51Z

functions/get_postgres_hosts.pp

+                    }
+                  }').map |$data| { $data['certname'] }
+  } else {
+    $postgres_hosts = []


$postgres_hosts would need to be returned in a format
puppet_metrics_dashboard:: postgres_host_list expects

So in this case, puppet_metrics_dashboard::postgres_host would need to be an array of elements, with each element being a postgres host. I've been looking at https://puppet.com/docs/puppet/7.0/lang_data_type.html

Initially, part of me thinks that I need to some thing like:

function rsan::get_postgres_hosts() { $postgres_hosts => Array #this is the bit im thinking of adding in $postgres_hosts = puppetdb_query('resources[certname] { type = "Class" and title = "Pe_postgresql::Server::Install" and nodes { deactivated is null and expired is null } }').map |$data| { $data['certname'] } }

My thinking is that by setting the data type beforehand - puppetdb_query will fill in the elements to that array as appropriate?

But I've also come across https://puppet.com/docs/puppet/7.0/lang_data_type.html#lang_data_type_usage

Where there is the 'Type' function. In this instance, I would do something like get the data type from the $postgres_hosts after the puppetdb_query is finished, then if the data type of $postgres_hosts =/ array, I set it as so?

MartyEwings · 2020-12-09T13:53:09Z

functions/get_puppet_servers.pp

@@ -0,0 +1,11 @@
+function rsan::get_puppet_servers() {
+  $puppet_servers =


there is a syntax issue here, best PDK validate

Syntax issue on this should be solved in my next commit from my local repo :)

MartyEwings · 2020-12-09T13:53:49Z

functions/get_puppet_servers.pp

@@ -0,0 +1,11 @@
+function rsan::get_puppet_servers() {
+  $puppet_servers =


$puppet_servers needs to be in a format

puppet_metrics_dashboard::master_list expects

MartyEwings · 2020-12-09T13:54:02Z

functions/get_puppetdb_hosts.pp

@@ -0,0 +1,17 @@
+function rsan::get_puppetdb_hosts() {
+  if $settings::storeconfigs {


same comments on the condition here

MartyEwings · 2020-12-09T13:59:30Z

functions/get_puppetdb_hosts.pp

+                    }
+                  }').map |$data| { $data['certname'] }
+  } else {
+    $puppetdb_hosts = []


puppetdb_hosts has to be returned in the correct format for

puppet_metrics_dashboard::puppetdb_list

m0dular · 2020-12-09T21:45:57Z

This functionality already exists in the enterprise_tasks repo, for examples finding postgres nodes here. Can we make that module a dependency of this project instead of rewriting functionality?

Also, the postgres query does not work because it doesn't capitalize each element of the class name.

MartyEwings · 2020-12-10T11:15:43Z

@m0dular Thanks for the input, that does look like a good solution.

The PRS here are for review and collaboration and are not meant to be a final code.

Would you like to added to the codeowners for review or added to the project?

Liam Sexton added 8 commits December 1, 2020 17:47

uncommented class and added instructions

48101d2

added function to search for db hosts

de69461

changed original function variable to $pdb

20757fe

reverted to just using class

89cfb85

functions for getting puppetdb hosts+puppet servers

abc93c1

created function for querting for postgres hosts

bbcc993

added function to query for puppet servers

d9e9c5b

added function calls, consisten variable names

49f3ed6

liamjohnsexton requested review from dylanratcliffe and MartyEwings as code owners December 8, 2020 11:12

Liam Sexton added 2 commits December 9, 2020 09:55

added comma to make pdk validate work

8f29421

changed ' to " to be safe in pdbquery

7ca726d

dylanratcliffe suggested changes Dec 9, 2020

View reviewed changes

MartyEwings requested changes Dec 9, 2020

View reviewed changes

changed the class title to be in capital letters

6ebf3e4

added deafult values for the class

2f0e788

liamjohnsexton changed the title ~~PR2: New functions created to query for each item - calling of functions in importer.pp~~ W.I.P. PR2: New functions created to query for each item - calling of functions in importer.pp Dec 15, 2020

MartyEwings added the enhancement New feature or request label Dec 17, 2020

MartyEwings added this to To Do in RSAN Development via automation Dec 17, 2020

MartyEwings added this to the Feature Complete - No VPN milestone Dec 17, 2020

MartyEwings linked an issue Dec 17, 2020 that may be closed by this pull request

Feature - Live Telemetry display #2

Closed

included postgresql::client

7e6dfb8

liamjohnsexton requested a review from jarretlavallee as a code owner January 7, 2021 10:56

liamjohnsexton added 5 commits January 7, 2021 11:13

put in correct comment line

41294cd

added if criteria

342737d

removed redefining aspect

3c5b46c

tricked you

749f73b

making pdk validate happy

fde7a15

liamjohnsexton added 4 commits January 7, 2021 12:10

msking pdk validate happy part 2

e1e7924

asdas

063f0c3

removing ruby code

f4dec00

altered syntax

92c9917

MartyEwings approved these changes Jan 7, 2021

View reviewed changes

Merge branch 'main' into metricfeature

9856f29

dylanratcliffe approved these changes Jan 7, 2021

View reviewed changes

MartyEwings merged commit 1f0d2f9 into main Jan 7, 2021

RSAN Development automation moved this from To Do to complete Jan 7, 2021

MartyEwings deleted the metricfeature branch January 11, 2021 15:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

W.I.P. PR2: New functions created to query for each item - calling of functions in importer.pp #10

W.I.P. PR2: New functions created to query for each item - calling of functions in importer.pp #10

liamjohnsexton commented Dec 8, 2020

dylanratcliffe left a comment

dylanratcliffe Dec 9, 2020

liamjohnsexton Dec 10, 2020

dylanratcliffe Dec 10, 2020

MartyEwings Dec 9, 2020 •

edited

MartyEwings Dec 9, 2020

liamjohnsexton Dec 10, 2020

MartyEwings Dec 9, 2020

liamjohnsexton Dec 10, 2020 •

edited

MartyEwings Dec 9, 2020

liamjohnsexton Dec 10, 2020

MartyEwings Dec 9, 2020

MartyEwings Dec 9, 2020

MartyEwings Dec 9, 2020

m0dular commented Dec 9, 2020

MartyEwings commented Dec 10, 2020


		#in the below class

		$puppet_severs = rsan::get_puppet_servers()

		@@ -0,0 +1,18 @@
		function rsan::get_postgres_hosts() {
		if $settings::storeconfigs {

		@@ -0,0 +1,11 @@
		function rsan::get_puppet_servers() {
		$puppet_servers =

		@@ -0,0 +1,17 @@
		function rsan::get_puppetdb_hosts() {
		if $settings::storeconfigs {

W.I.P. PR2: New functions created to query for each item - calling of functions in importer.pp #10

W.I.P. PR2: New functions created to query for each item - calling of functions in importer.pp #10

Conversation

liamjohnsexton commented Dec 8, 2020

dylanratcliffe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MartyEwings Dec 9, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liamjohnsexton Dec 10, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

m0dular commented Dec 9, 2020

MartyEwings commented Dec 10, 2020

MartyEwings Dec 9, 2020 •

edited

liamjohnsexton Dec 10, 2020 •

edited