Monitoring system performance, activity & email alerts
What ever your setup, once running, you should monitor your resource usage, as running out of CPU will cause trunk recorder to drop voice signals and create garble/noises. You can see this happening in the console with printing of OOOOOO's FAQ#i-am-seeing-a-lot-of-0000s-or-oooos-in-my-console And you will need to either upgrade your system or reduce allocated bandwidth/recorders.
The simplest way is just with "top" command, where you can see your load average over 1,5 & 15 minute average as well as momentary cpu usage. CPU usage is 100% per cpu core and load is 1.0 per core, so 4 core will maxed out at 4.0 load and 400% usage, but you'll start to see OOOO's before that.
Another simple terminal program that can show this (load) over time in a graph is 'ttyload' but it will only show you history while you're running it.
For longer term monitoring you can use 'munin' which stores records/graphs over day/week/month for all system resources. (keep in mind with this though that it currently logs at 5 minute interval so it wont necessarily catch intermittent high load generated during a call recording if your system is not very active, this is where watching live in terminal while a recording is taking place is still necessary.
You should also and likely will automatically check the output recordings as there may be weak signal or other problems, they can be optionally deleted or not.
Here are some rough notes from getting munin working under debian,
install munin and apache2 packages
edit /etc/munin/munin.conf uncommenting the following line at top to enable web content generation
dbdir /var/lib/munin
htmldir /var/cache/munin/www
logdir /var/log/munin
rundir /var/run/munin
# Where to look for the HTML templates
#
tmpldir /etc/munin/templates
sudo systemctl restart munin
ln -s /etc/munin/apache24.conf /etc/apache2/conf-enabled/munin.conf
sudo systemctl restart apache2
open http://device_ip/munin in browser
Munin comes default with a bunch of plugins installed & enabled, so you should get graphs generated within 5-10mins, plugins/graphs can be toggled with symlinks/files in /etc/munin/plugins/ followed be a service restart. (systemctl munin-node restart)
It works by using uploadScript setting to copy the json call details file to a chosen folder, then the munin plugin processes those json files for data to graph. Currently it just graphs total calls per hour rate, though the time rate is adjustable depending on you activity levels.
Create the following file and set to permissions 744
#!/usr/bin/env python3
#/etc/munin/plugins/trunkrecorder-munin.py
import os
import sys
import datetime
import json
import time
timeSpan = 3600 #seconds, call rate per
title = "Trunk Recorder Calls/Hour"
logDir = '/var/log/munin/trunkrecorder-munin'
### temp storage between runs, incase next run doesnt have listener count
tgStore = open("/var/log/munin/trunkrecorder-munin/talkgroups.txt", "a+")
tgStore.seek(0)#needed as append puts pointer at end
try:
talkGroups = json.load(tgStore)
except:
talkGroups = {}
totalCount = 0
for filename in os.listdir(logDir):
if 'json' in filename:
jsonpath = logDir+'/'+filename
with open(jsonpath, 'r') as f:
try:
jsonData = json.load(f)
except:
continue
talkgroup = str(jsonData['talkgroup'])#need as str as becoming dict key
if(time.time() - jsonData['start_time'] > timeSpan): #1 hour (calls per hour)
os.remove(jsonpath)
else:
totalCount+=1
if talkgroup in talkGroups:
talkGroups[talkgroup]['count']+=1
talkGroups[talkgroup]['talkgroup_tag'] = jsonData['talkgroup_tag']#incase tag changes
else:
talkGroups[talkgroup] = {'talkgroup_tag': jsonData['talkgroup_tag'], 'count': 1 }
#sort talkGroups dict
talkGroups = {i:talkGroups[i] for i in sorted(talkGroups.keys())}
if('config' in sys.argv):
print('graph_title '+title)
print('graph_vlabel Call Count')
print('graph_printf %.0lf' )
print('graph_category trunkrecorder')
print('total.label Total')
print('total.draw AREA')
print('total.colour e3e3e3')
print('total.critical 1:') #alert if falls to 0, email config must be setup too
for id, data in talkGroups.items():
#print(id +'.draw AREASTACK')
print(id +'.label '+ id +':'+data['talkgroup_tag'])
sys.exit()
print("total.value " + str(totalCount))
for id, data in talkGroups.items():
print(str(id)+'.value '+str(data['count']))
### update temp storage
#set count to 0
for id in talkGroups:
talkGroups[id]['count'] = 0
#set count to 0
tgStore.seek(0)
tgStore.truncate()
tgStore.write(json.dumps(talkGroups))
tgStore.close()
###
then add the following to bottom of /etc/munin/plugin-conf.d/munin-node
[trunkrecorder.py]
user root
group root
restart munin
sudo systemctl restart munin-node
to see if plugin enabled run:
munin-node-configure
to test run plugin to see if working:
munin-run trunkrecorder
Here is the python code that must be called by uploadScript feature of trunk recorder, either add it to your existing python script or add one
##############################
#log for munin
logDir = '/var/log/munin/trunkrecorder-munin'
if not os.path.exists(logDir):
os.makedirs(logDir)
#copyjson data file for processing by munin plugin
import shutil
shutil.copy(jsonpath, logDir+'/'+ os.path.split(jsonpath)[1])
#log for munin
###############################
You must also add the following mount to your trunk recorder docker container if using docker so the created files end up on the host and not just in the container:
- '/var/log/munin:/var/log/munin'
Munin can optionally have alarms set that can trigger for example an email. I use swaks package to send the mail via a smtp account i have. Edit /etc/munin/munin.conf :
contact.email.command swaks --to destination@email.com --from source@email.com --header 'Subject: Munin-notification for ${var:plugin}' --body 'Current value of ${var:label} is ${var:value}' --server smtp.server.com.au:587 --auth LOGIN --auth-user "source@email.com" --auth-password "password" -tls
the alerts levels by default are set in the plugin file eg
print('trunk.critical 1:') #alert if falls to 0 to warn me that something likely broke
I found the following command to send a test mail on demand:
su - munin --shell=/bin/bash -c "/usr/share/munin/munin-limits --contact email --force"
more details:
http://guide.munin-monitoring.org/en/latest/tutorial/alert.html
http://guide.munin-monitoring.org/en/latest/reference/plugin.html#fieldname-critical