[WIP] Integrating Monit for monitoring TorQ #105

rdanutalexandru1993 · 2018-07-17T13:00:28Z

This will add the basic configuration needed to use monit to monitor TorQ processes on UNIX-like systems. The processes will be read from the process.csv file, started and monitored by monit. The dashboard can be accessed on localhost port 2812 and will include basic information about the TorQ processes. If any of the processes fail to start, monit will try to restart the process and send an alert e-mail which contains information about the service.

Created the backbone of monit.sh

…om process.csv Added monitconfig.cfg which contains the location of the process.csv and the output file

Modified the location of the files Updated the monitrc to include the monitconfi.cfg

Added a new test alert in monitconfig.cfg which executest a test bash script Modified the e-mail format in /code/templates/monitrc to include more details about the event

jonathonmcmurray · 2018-07-18T10:56:55Z

code/scripts/templates/monitrc

+#                backup.bar.baz port 10025,  # backup mailserver on port 10025
+#                localhost                   # fallback relay
+  set mailserver smtp.gmail.com port 587
+        username "\"torqmonit@gmail.com\"" password "\"torqmonit2018\""


is this a real email address & password? these should be dummy placeholders in here, and this password is now in git history so should also be changed

It is an e-mail address that I will change. I only use this to test if monit works as it should. I wasn't able to make the e-mail work yet.

Yeah, but regardless you should never commit a password to any email account to git - right now anyone could log into that email account (and I did, to check)

jonathonmcmurray · 2018-07-18T10:57:25Z

code/scripts/templates/monitrc

+## commands to a running Monit daemon. See the Monit Wiki if you want to
+## enable SSL for the HTTP interface.
+#
+ set httpd port 2812 and


any way to have this port based on TorQ baseport variable? what happens if a user already has something running on port 2812?

this is the default port for monit. It wasn't my focus to change it, but it should be really easy to do so. monitrc is created using the environment variables which are set when setenv.sh is executed. Should I look at a way to set that port taking in consideration the baseport and the number of processes or should I maybe define a an env variable and let the user set the port (just like we do now with the baseport)

personally I think the less a user has to think about configuring the better, so tying it to the baseport seems most logical to me

jonathonmcmurray · 2018-07-18T10:58:42Z

code/scripts/templates/testmail.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+echo -e "TORQMONIT\r\rThis is an automated message\r\rProcess SERVICE is down"|mail.mailutils -s \"SUBJECT\" rdanutalexandrugmail.com


I assume this file should be removed? Doesn't seem like something that should be merged to master

This is my attempt to use an external e-mail to generate the alerts. Not very successful so far. If it works this will take the form of a template e-mail that I will use in my monitalert.cfg
`

jonathonmcmurray · 2018-07-18T10:59:30Z

setenv.sh

@@ -12,8 +12,10 @@ export KDBAPPCODE=${TORQAPPHOME}/code
 export KDBHDB=${TORQDATA}/hdb/database
 export KDBWDB=${TORQDATA}/wdbhdb

-export KDBBASEPORT=6000                                                                             # set KDBBASEPORT to the default value for a TorQ Installation
+export KDBBASEPORT=6000                                                                            # set KDBBASEPORT to the default value for a TorQ Installation


looks like alignment changed on this comment bringing it out of line, please put it back

jonathonmcmurray · 2018-07-18T11:00:58Z

fill_templates.sh

+    dirpath="$0"
+fi
+
+eval ". $(dirname "$dirpath")/setenv.sh"                                                            #set environment variables  


why is eval needed here? Why not just source the file with . directly? (this pattern exists in a few places)

This was already here. I think eval is used to execute ". $(dirname "$dirpath")/setenv.sh"

Not sure what you mean by "already here", this is a new file, no? I get what eval is doing, I'm asking why though? Seems like you could just do . $(dirname "$dirpath")/setenv.sh

tsmyth-aquaq · 2018-07-23T08:54:47Z

code/scripts/templates/monitconfig.cfg

@@ -0,0 +1 @@
+${TORQHOME}/appconfig/process.csv     ${TORQHOME}/torq.sh     monitconfig.cfg


This + fill_templates.sh was set up by me a long time ago rather naively, there's probably better ways to do it and definitely better ways to set up this config file, would be worth looking into it.

…e status output Modified torq.sh to setenv using a BASEDIR defined in the .bashrc_torq Enabled the log file output in monitrc Modified monit.sh to create a .bashrc_torq which holds the environment variables necessary for starting torq from root

rdanutalexandru1993 · 2018-08-07T16:53:08Z

bash monit.sh takes me in the root directory and tries to execute everything from there. This creates a number of issues. I have seen someone using start program = "/bin/bash -c 'cd location && ./torq.sh $procname'". At the moment I am defining TORQHOME in a file called .bashrc_torq and source it so I can use the full path. Anyone knows a better way for doing solving this problem?

@ThomasSmyth @jonathonmcmurray @jemmaborland @claire-sloan @jonnypress

jonathonmcmurray · 2018-08-07T17:55:09Z

monit.sh

+  checkst "echo $TORQHOME|wc -l" "TORQHOME present" "npres" "1"
+  checkst "echo $BASEDIR|wc -l" "BASEDIR present" "npres" "1"
+else
+  echo "export TORQHOME=${PWD}" >> $HOME/.bashrc_torq


Does all this stuff need to be duplicated? Can you not just move it out of the if statement & run it regardless? Or is there a difference I'm missing?

jonathonmcmurray · 2018-08-07T17:58:17Z

@Aqualex I don't think I understand the problem...why can't you just source setenv.sh the same way the TorQ start script does and get TORQHOME defined that way? Seems really messy to write a second script and then source it, plus the name .bashrc_torq doesn't really make sense; .bashrc is a file loaded by bash at the start of a shell session, that's not really related to this. I don't get why it's necessary to write a second script, if the purpose is to set environment variables, can you just do that directly if required? Why write into a second script & source it instead of just doing it directly?

… the path to torq.sh and process.csv from fill_templates.sh Modified the way the path is set in templates/monitrc to include the full path not only the path relative to the current directory Modified e-mail settings in monitrc from ssl to tslv1 and fixed the issues with sending alert e-mails Added path to events folder in templates/monitrc Fixed e-mail format in templates/monitrc Changed port for HTTP interface Fixed path to torq.sh in fill_templates Modified monit.sh to run fill_templates to run if any of the necessary files are missing Modified fill_templates to check if the necessary files exist and create them only it they are missing Modified path to setenv.sh in torq.sh to be set based on the current directory Added small script in torq.sh, monit.sh to determine the path based on the current directory

…rectory the script is executed from

Fixed format for e-mail template Removed personal e-mail from monitalert template

jonnypress · 2018-08-09T09:03:19Z

config/settings/gateway.q

@@ -11,5 +11,5 @@ clearinactivetime:0D01:00	// the time to keep inactive handle data

 // Server connection details
 \d .servers
-CONNECTIONS:`rdb`hdb		// list of connections to make at start up
+CONNECTIONS:`rdb`hdb`wdb		// list of connections to make at start up


why is this here?

jonnypress · 2018-08-09T09:04:14Z

fill_templates.sh

@@ -0,0 +1,88 @@
+#!/bin/bash


I'd rather this script didn't sit at the top level directory of TorQ

…reated and how monit is started

Added usage function in monit.sh Modified monitrc to used the BASEDIR instead of PWD

jonnypress

Can you walk me through this on Thursday please?

jonnypress · 2018-08-14T15:53:08Z

monit/bin/monit.sh

@@ -0,0 +1,162 @@
+#!/bin/bash
+#FUNTION DECLARATION ###############################################################################


jonnypress · 2018-08-14T15:58:47Z

monit/templates/monitalert.cfg

@@ -0,0 +1,12 @@
+check system homer.aquaq.co.uk 


take our server name out
And the google check
Best to specify that this is an example

jonnypress · 2018-08-14T15:59:45Z

monit/templates/monitrc

@@ -0,0 +1,321 @@
+###############################################################################


Don't want this in here. All we want to create is the monit config file. Leave it up to the person using it to add it to their monitrc. Otherwise we become coupled to changes in monitrc

so you want me to remove this altogether and everything the bit in monit.sh that creates it? I believe we should have monit setup in such way that it has some basic functionality straigth out of the box so that our users will see that it works at least.

jonnypress · 2018-08-14T16:00:42Z

torq.sh

@@ -221,8 +221,14 @@ usage() {
  exit 1


Why do you need to change torq.sh in this PR?

monit executes torq.sh from the root directory. Torq.sh uses . setenv.sh and obviously the setenv.sh is not in the root directory

jonnypress · 2018-08-14T16:00:55Z

setenv.sh

-export TORQHOME=${PWD}                                                                              # if running the kdb+tick example, change these to full paths
-export TORQDATA=${PWD}                                                                              # some of the kdb+tick processes will change directory, and these will no longer be valid
-export TORQAPPHOME=${PWD}
+if [ "-bash" = $0 ]; then


Are these changes strictly for this PR?

jonnypress · 2018-08-24T09:38:26Z

As per conversation, add some documentation and we are good to go. We can start with basic documentation and flesh it out.

ADDED an empty documentation file

Removed unnecessary comment

rdanutalexandru1993 added 6 commits July 6, 2018 15:08

First attempt at the monitoring directory structure

91799bd

Created the backbone of fills_templates.sh

695bca6

Created the backbone of monit.sh

Added function createmonitrc which takes the procname and proctype fr…

e95611d

…om process.csv Added monitconfig.cfg which contains the location of the process.csv and the output file

Modified monittemplate.txt to use the new TorQ start script

17e46de

Modified the location of the files Updated the monitrc to include the monitconfi.cfg

Removed unnecessary quotes after createmonconfig function

75bd90e

Added new alerts in monitalert.cfg to monitor the host

355b13d

Added a new test alert in monitconfig.cfg which executest a test bash script Modified the e-mail format in /code/templates/monitrc to include more details about the event

rdanutalexandru1993 added the enhancement label Jul 17, 2018

rdanutalexandru1993 self-assigned this Jul 17, 2018

rdanutalexandru1993 requested review from jonnypress, tsmyth-aquaq, claire-sloan, jonathonmcmurray and jemmaborland July 17, 2018 13:00

rdanutalexandru1993 changed the title ~~[WIP] Monit for monitoring TorQ~~ [WIP] Integrating Monit for monitoring TorQ Jul 17, 2018

Attempted to add email alerts using mail.mailutils

ee1fc5c

jonathonmcmurray reviewed Jul 18, 2018

View reviewed changes

Merge branch 'master' into monit

3639a6e

tsmyth-aquaq reviewed Jul 23, 2018

View reviewed changes

rdanutalexandru1993 added 2 commits August 6, 2018 14:45

Merge remote-tracking branch 'origin/master' into monit

f770e3e

rdanutalexandru1993 added the help wanted label Aug 7, 2018

jonathonmcmurray reviewed Aug 7, 2018

View reviewed changes

rdanutalexandru1993 added 5 commits August 8, 2018 14:53

Modified setenv.sh to set the path for Torq direcotry based on the di…

2dcc6c1

…rectory the script is executed from

Fixed spacing for comment

1aeb204

Removed unnecessary testmai.sh file

ff5d14e

Removed personal e-mail, username and password from the template monitrc

cb93dd2

Fixed format for e-mail template Removed personal e-mail from monitalert template

jonnypress reviewed Aug 9, 2018

View reviewed changes

rdanutalexandru1993 added 4 commits August 9, 2018 17:34

Modified monit.sh to be more flexible with how the config files are c…

d3e7f85

…reated and how monit is started

Removed old monit.sh

45ea5a3

Added usage function in monit.sh Modified monitrc to used the BASEDIR instead of PWD

Reverted a change in gateway.q

9d82a96

Removed unnecessary monitconfig.cfg

dbd21ff

jonnypress reviewed Aug 14, 2018

View reviewed changes

rdanutalexandru1993 added 2 commits August 16, 2018 15:52

Removed homer.aquaq.co.uk and google.com from the PR

b468b3b

Fixed typo

add361b

rdanutalexandru1993 removed the help wanted label Aug 24, 2018

rdanutalexandru1993 and others added 6 commits September 3, 2018 19:27

Merge remote-tracking branch 'origin/master' into monit

cef0a1b

FIXED error with certificates in setenv.sh

8d0c433

ADDED an empty documentation file

Added documentation for monit

3c36469

Removed unnecessary comment

Fixed indentation for nested list

cb05f05

Minor modifications to wording

54dbcaa

added monit to menu

396ee30

jonnypress approved these changes Sep 4, 2018

View reviewed changes

jonnypress merged commit 41b382e into master Sep 4, 2018

jonnypress deleted the monit branch September 4, 2018 11:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Integrating Monit for monitoring TorQ #105

[WIP] Integrating Monit for monitoring TorQ #105

rdanutalexandru1993 commented Jul 17, 2018

jonathonmcmurray Jul 18, 2018

rdanutalexandru1993 Jul 18, 2018

jonathonmcmurray Jul 18, 2018

jonathonmcmurray Jul 18, 2018 •

edited

rdanutalexandru1993 Jul 18, 2018

jonathonmcmurray Jul 18, 2018

jonathonmcmurray Jul 18, 2018

rdanutalexandru1993 Jul 18, 2018 •

edited

jonathonmcmurray Jul 18, 2018

rdanutalexandru1993 Jul 18, 2018

jonathonmcmurray Jul 18, 2018

rdanutalexandru1993 Jul 18, 2018

jonathonmcmurray Jul 18, 2018

tsmyth-aquaq Jul 23, 2018

rdanutalexandru1993 commented Aug 7, 2018 •

edited

jonathonmcmurray Aug 7, 2018

jonathonmcmurray commented Aug 7, 2018

jonnypress Aug 9, 2018

jonnypress Aug 9, 2018

jonnypress left a comment

jonnypress Aug 14, 2018

rdanutalexandru1993 Aug 16, 2018

jonnypress Aug 14, 2018

rdanutalexandru1993 Aug 16, 2018

jonnypress Aug 14, 2018

rdanutalexandru1993 Aug 16, 2018

jonnypress Aug 14, 2018

rdanutalexandru1993 Aug 16, 2018

jonnypress Aug 14, 2018

rdanutalexandru1993 Aug 16, 2018

jonnypress commented Aug 24, 2018

		@@ -0,0 +1,3 @@
		#!/bin/bash

		echo -e "TORQMONIT\r\rThis is an automated message\r\rProcess SERVICE is down"\|mail.mailutils -s \"SUBJECT\" rdanutalexandrugmail.com

		@@ -0,0 +1 @@
		${TORQHOME}/appconfig/process.csv ${TORQHOME}/torq.sh monitconfig.cfg

		@@ -0,0 +1,162 @@
		#!/bin/bash
		#FUNTION DECLARATION ###############################################################################

		@@ -0,0 +1,321 @@
		###############################################################################

[WIP] Integrating Monit for monitoring TorQ #105

[WIP] Integrating Monit for monitoring TorQ #105

Conversation

rdanutalexandru1993 commented Jul 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathonmcmurray Jul 18, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdanutalexandru1993 Jul 18, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdanutalexandru1993 commented Aug 7, 2018 • edited

Choose a reason for hiding this comment

jonathonmcmurray commented Aug 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonnypress left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonnypress commented Aug 24, 2018

jonathonmcmurray Jul 18, 2018 •

edited

rdanutalexandru1993 Jul 18, 2018 •

edited

rdanutalexandru1993 commented Aug 7, 2018 •

edited