Better instructions and scripts for example-base

example-base/README

@@ -7,7 +7,52 @@ This folder contains example szarp_server base, that can be run on any computer
 Remember that scripts given here should not be used or modified to violate rules or laws of services or libaries they use. Newterm takes no responsibility for how they are used.
 
 Scripts given here are only simple example of how a single user can use our software. Showing all available options would create document far exceeding readable example borders. If you're curious about other uses of our software feel free to check our documentation located at:
-	https://szarp.org//documentation/new/ipk/html/szarp/html
+	https://szarp.org//documentation/new/ipk/html/szarp.html
 
 If you want to contribute to our project checkout our Github website :
 	https://github.com/Newterm/szarp
+
+First time using SZARP :
+
+1. Start with package installation as it will automatically prepare your environment for using SZARP software. Follow instruction on https://szarp.org/en/download/ and install packages : szarp-server and szarp-wx.
+
+ During installation you will be prompted about 3 things:
+	* database type :
+		- sz3 - our old database type, using meaner3 to save data, every sample is 16-bit integer
+		- sz4 - recommended database type, using meaner4 to save data, allows using additional API for saving different data types
+		- sz3+sz4 - both sz3 and sz4 running simultaneously
+	* creating szbase catalog - pick yes, as without szbase catalog located inside : "/opt/szarp/$base_prefix/szbase" meaners won't work. Picking "no" only for advanced users.
+	* default base prefix - prefix of your new database, it is recommended to name it the same as your hostname, as specific SZARP package, as a safety measure may stop SZARP software from running if default database name and hostname don't match
+	CAREFUL! leaving default base prefix blank causes undefined behaviour and may permanently break your SZARP installation.
+	* after successful installation you should be able to find your installation in : /opt/szarp/$prefix/ and all SZARP binaries in /opt/szarp/bin
+2. Prepare your example base configuration:
+	* start with configuring your daemons. Inside this folder you will find numerous examples of python scripts compatible with pythondmn. Each running script needs to be configured and must have corresponding device configured inside config/params.xml file. Check pdmn_sample.py for more comments on creating python scripts and config/params.xml to get a feel on how a proper configuration should look like.
+	* run your szarp-server with :
+		systemctl start szarp-server.target ( or restart if its already running )
+	* to check if szarp is working grep running processes for szarp :
+		ps aux | grep szarp
+	* example running sz3 base :
+$ ps aux | grep szarp
+root      6975  0.0  0.1 179932 11148 ?        Ssl  12:19   0:00 /opt/szarp/bin/cfgdealer
+root      6976  0.0  0.2 173156 16312 ?        Ssl  12:19   0:00 /usr/bin/python2 /opt/szarp/lib/python/parhub.py --no-daemon
+root      6977  0.0  0.2 221600 20268 ?        Ssl  12:19   0:00 /opt/szarp/bin/parcook -n
+root      6979  0.0  0.1 188932 15212 ?        Ss   12:19   0:00 /opt/szarp/bin/meaner3 -n
+root      6981  0.0  0.2 185332 16240 ?        Ss   12:19   0:00 /opt/szarp/bin/sender --no-daemon
+root      6994  0.0  0.0 125088  6680 ?        S    12:19   0:00 /opt/szarp/bin/paramd
+root      6995  1.6  0.3 313504 26288 ?        Sl   12:19   0:00 /opt/szarp/bin/pythondmn 1 /opt/szarp/example/pdmn_git.py
+root      6999  0.0  0.0 125088  6680 ?        S    12:19   0:00 /opt/szarp/bin/paramd
+root      7000  0.0  0.0 125088  6680 ?        S    12:19   0:00 /opt/szarp/bin/paramd
+root      7004  0.0  0.3  83108 27196 ?        S    12:19   0:00 /usr/bin/python /usr/bin/twistd -y /opt/szarp/lib/python/probes_server.tac --logfile - --pidfile /var/run/probes_server.pid
+user   10213  0.0  0.0  12784   984 pts/1    S+   12:20   0:00 grep --color=auto szarp
+
+	* after leaving your daemons for 10 minutes you should be able to see your collected data. Run
+		/opt/szarp/bin/draw3
+	to see your data. For tutorials on how to use draw3 check :
+		https://www.youtube.com/watch?v=kdNZSWAP3zQ&t=119s
+		https://www.youtube.com/watch?v=NKdkN01Ig60&t=8s
+		https://www.youtube.com/watch?v=jktQu56L9Fk&t=161s
+
+To use any of suplemented scripts remember to change their base prefix inside main function and configuration files.
+
+Every vailable pdmn script contains its short description and changes needed to run it. To get better hang of how SZARP works try to write your own scripts and test them. To see more detailed comments about different scripts sections check out pdmn_sample as it holds numerous examples and comments on how to create new scripts. If you feel confident in the usefulness of your creations you can also send us a Pull Request and we might incorporate them into this example. Good luck!
+

example-base/config/params.xml

@@ -129,10 +129,6 @@
 	<device daemon="/opt/szarp/bin/pythondmn" path="/opt/szarp/example-base/pdmn_cpu.py" parcook_device="no">
 		<unit id="1" type="1" subtype="1" bufsize="1">
 		   <!-- For sz4 daemons you can specify data_type to simplify values loading. -->
-		  <param name="System:CPU:Running processes" short_name="RP" unit="-" prec="0" base_ind="auto" data_type="integer">
-			<raport title="System report" order="1"/>
-			<draw title="Processor" min="0" max="500" order="1"/>
-		  </param>
 		   <param name="System:CPU:Core 1 usage" short_name="used" unit="%" prec="1" base_ind="auto" data_type="float">
 			<raport title="System report" order="2"/>
 			<draw title="Processor" min="0" max="100" order="2"/>
@@ -148,6 +144,10 @@
 		   <param name="System:CPU:Core 4 usage" short_name="used" unit="%" prec="1" base_ind="auto" data_type="float">
 			<raport title="System report" order="5"/>
 			<draw title="Processor" min="0" max="100" order="2"/>
+		  </param>
+		  <param name="System:CPU:Running processes" short_name="RP" unit="-" prec="0" base_ind="auto" data_type="integer">
+			<raport title="System report" order="1"/>
+			<draw title="Processor" min="0" max="500" order="1"/>
 		  </param>
 		   <param name="System:Memory:Load" short_name="load" unit="-" prec="2" base_ind="auto" data_type="double">
 			<raport title="System report" order="5"/>

example-base/pdmn_cpu.py

@@ -3,44 +3,44 @@
 # script to monitor system resources usage
 # shows cores usage, running processes and load
 # cpu_usage len is different on each processor so remember to change accordingly!
+# requires psutil to work
+# insert your threads number as NUMBER_OF_THREADS to check CPU usage
+# remember to change your params.xml accordingly
 
 import sys                                  # exit()
 import time                                 # sleep()
-import logging                              # logowanie (istotne!)
-from logging.handlers import SysLogHandler  #
-sys.path.append("/opt/szarp/lib/python")    #
-from pydaemontimer import ReTimer           # opcjonalnie zamiast sleepa
+import logging
+from logging.handlers import SysLogHandler
+sys.path.append("/opt/szarp/lib/python")
+from pydaemontimer import ReTimer
 
 import psutil
 import os
 import subprocess
 from subprocess import Popen, PIPE
 
+BASE_PREFIX = ""
+NUMBER_OF_THREADS = 4
+
 class CPUReader:
 	def get_running_processes(self):
 		p = subprocess.Popen("ps ux | wc -l", shell=True,stdout=PIPE)
-		if p.wait():
-			raise Exception("failed to get running processes")
 		return int(p.communicate()[0].strip())
 
 	def update_values(self):
 		try:
 			rp = self.get_running_processes()
 			cpu_usage = psutil.cpu_percent(percpu=True)
-			ipc.set_read_sz4_int(0, rp)
-			ipc.set_read_sz4_float(1, cpu_usage[0] )
-			ipc.set_read_sz4_float(2, cpu_usage[1] )
-			ipc.set_read_sz4_float(3, cpu_usage[2] )
-			ipc.set_read_sz4_float(4, cpu_usage[3] )
-			ipc.set_read_sz4_double(5, os.getloadavg()[0] )
+			for i in xrange(0, NUMBER_OF_THREADS):
+				ipc.set_read_sz4_float(i, cpu_usage[i])
+			ipc.set_read_sz4_int(NUMBER_OF_THREADS, rp)
+			ipc.set_read_sz4_double(NUMBER_OF_THREADS+1, os.getloadavg()[0] )
 		except Exception as err:
 			logger.error("failed to fetch data, %s", str(err).splitlines()[0])
-			ipc.set_no_data(0)
-			ipc.set_no_data(1)
-			ipc.set_no_data(2)
-			ipc.set_no_data(3)
-			ipc.set_no_data(4)
-			ipc.set_no_data(5)
+			for i in xrange(0, NUMBER_OF_THREADS):
+				ipc.set_no_data(i)
+			ipc.set_no_data(NUMBER_OF_THREADS)
+			ipc.set_no_data(NUMBER_OF_THREADS+1)
 
 def main(argv=None):
 	psutil.cpu_percent(percpu=True)
@@ -57,7 +57,7 @@ def main(argv=None):
 		global ipc
 		sys.path.append('/opt/szarp/lib/python')
 		from test_ipc import TestIPC
-		ipc = TestIPC("example-base", "/opt/szarp/example-base/pdmn_cpu.py")
+		ipc = TestIPC("%s"%BASE_PREFIX, "/opt/szarp/%s/pdmn_cpu.py"%BASE_PREFIX)
 
 	ex = CPUReader()
 

example-base/pdmn_git.py

@@ -5,6 +5,8 @@
 #allows to track commits, lines and files counter
 #checks each minute number of lines and files
 #and every hour number of commits
+#fill REPO_PARH and BRANCH name to run this script
+#change suffixes if you want to include more files to check
 
 import sys                                  # exit()
 import time                                 # sleep()
@@ -13,6 +15,8 @@
 import subprocess
 from subprocess import PIPE, Popen
 
+BASE_PREFIX = ""
+
 suffixes = [ "cpp", "h", "txt", "hpp", "py", "cc" ]
 
 REPO_PATH = "absolute_repository_path"
@@ -24,25 +28,19 @@ def get_repo_lines(self, path):
 		if path.endswith("/") :
 			path = path[:-1]
 		p = subprocess.Popen("git -C %s ls-files" % path, shell=True,stdout=PIPE)
-		if p.wait():
-			raise Exception("getting repo lines failed")
 		files = p.communicate()[0].strip().split('\n')
 		counter = 0
 		files_counter = 0
 		for f in files :
 			if not f.split(".")[-1] in suffixes:
 				continue
 			lines = subprocess.Popen("cat %s/%s | wc -l"%(path,f),shell=True, stdout=PIPE)
-			if lines.wait():
-				raise Exception("getting lines from file failed")
 			counter += int(lines.communicate()[0].strip())
 			files_counter += 1
 		return (counter, files_counter)
 
 	def get_repo_commits(self, path, branch):
 		p = subprocess.Popen("git -C %s rev-list --count %s" % (path, branch), shell=True, stdout=PIPE)
-		if p.wait():
-			raise Exception("getting repo commits failed")
 		return int(p.communicate()[0].strip())
 
 	def update_commits(self):
@@ -80,7 +78,7 @@ def main(argv=None):
 		global ipc
 		sys.path.append('/opt/szarp/lib/python')
 		from test_ipc import TestIPC
-		ipc = TestIPC("example-base", "/opt/szarp/example-base/pdmn_git.py")
+		ipc = TestIPC("%s"%BASE_PREFIX, "/opt/szarp/%s/pdmn_git.py"%BASE_PREFIX)
 
 	ex = RepoReader()
 	time.sleep(10)

example-base/pdmn_instagram.py

@@ -2,6 +2,7 @@
 # -*- encoding: utf-8 -*-
 
 #script to gather and show data from Instagram
+#change photo_URL and account_URL to run this script
 
 import sys                                  # exit()
 import time                                 # sleep()
@@ -11,6 +12,7 @@
 import requests
 import re
 
+BASE_PREFIX = ""
 photo_URL = "URL_to_instagram_photo"
 account_URL = "URL_to_instagram_account"
 
@@ -60,6 +62,7 @@ def update_values(self):
 			ipc.set_no_data(1)
 			ipc.set_no_data(2)
 			ipc.set_no_data(3)
+			ipc.set_no_data(4)
 
 		ipc.go_parcook()
 
@@ -77,7 +80,7 @@ def main(argv=None):
 		global ipc
 		sys.path.append('/opt/szarp/lib/python')
 		from test_ipc import TestIPC
-		ipc = TestIPC("example-base", "/opt/szarp/example-base/pdmn_instagram.py")
+		ipc = TestIPC("%s"%BASE_PREFIX, "/opt/szarp/%s/pdmn_instagram.py"%BASE_PREFIX)
 
 	ex = InstaReader()
 	time.sleep(10)

example-base/pdmn_ram.py

@@ -3,13 +3,15 @@
 #python-psutil
 
 # simple python script to get RAM usage
+# ready to run and gather data
 
 import sys                                  # exit()
 import time                                 # sleep()
-import logging                              # logowanie (istotne!)
+import logging
 import psutil
 from logging.handlers import SysLogHandler  #
 
+BASE_PREFIX = ""
 MiB = 1024 * 1024
 
 class RAMReader:
@@ -41,7 +43,7 @@ def main(argv=None):
 		global ipc
 		sys.path.append('/opt/szarp/lib/python')
 		from test_ipc import TestIPC
-		ipc = TestIPC("example-base", "/opt/szarp/example-base/pdmn_ram.py")
+		ipc = TestIPC("%s"%BASE_PREFIX, "/opt/szarp/%s/pdmn_ram.py"%BASE_PREFIX)
 
 	ex = RAMReader()
 

example-base/pdmn_sample.py

@@ -10,6 +10,8 @@
 import logging                              # logging ( important ! )
 from logging.handlers import SysLogHandler
 
+BASE_PREFIX = ""
+
 sys.path.append("/opt/szarp/lib/python")    # use with sz4 python functions
 #from pydaemontimer import ReTimer           # optional instead of sleep
 
@@ -80,7 +82,7 @@ def main(argv=None):
 		global ipc
 		sys.path.append('/opt/szarp/lib/python')
 		from test_ipc import TestIPC
-		ipc = TestIPC("example-base", "/opt/szarp/example-base/pdmn_sample.py")
+		ipc = TestIPC("%s"%BASE_PREFIX, "/opt/szarp/%s/pdmn_sample.py"%BASE_PREFIX)
 
 	#create data reader instance
 	ex = RAMReader()

example-base/pdmn_weather.py

@@ -5,6 +5,7 @@
 #uses pyowm libary to simplify observation requests
 #to use this script you need your own OWM_TOKEN
 #you can test location availability on the given site
+#fill OWM_TOKEN and LOCATION to run this script
 
 #python-owm
 #https://github.com/csparpa/pyowm
@@ -13,11 +14,12 @@
 
 import sys                                  # exit()
 import time                                 # sleep()
-import logging                              # logowanie (istotne!)
-from logging.handlers import SysLogHandler  #
+import logging
+from logging.handlers import SysLogHandler
 
 OWM_TOKEN = 'you_OWM_token'
 LOCATION = 'Warsaw,PL'
+BASE_PREFIX = ""
 
 class WeatherReader:
 	def __init__(self):
@@ -60,7 +62,7 @@ def main(argv=None):
 		global ipc
 		sys.path.append('/opt/szarp/lib/python')
 		from test_ipc import TestIPC
-		ipc = TestIPC("example-base", "/opt/szarp/example-base/pdmn_weather.py")
+		ipc = TestIPC("%s"%BASE_PREFIX, "/opt/szarp/%s/pdmn_weather.py"%BASE_PREFIX)
 
 	ex = WeatherReader()
 

example-base/pdmn_youtube.py

@@ -7,14 +7,23 @@
 
 #uses pafy python libary to simplify youtube requests
 #details : https://github.com/mps-youtube/pafy
+#requires pafy and youtube-dl to be installed
+#easiest way is to do this with pip
+#fill extra:url in config/params.xml to gather data with this script
+
 import pafy
 
 import sys                                  # exit()
 import time                                 # sleep()
-import logging                              # logowanie (istotne!)
+import logging                              # logging (important!)
+sys.path.append("/opt/szarp/lib/python")    #
+from pydaemontimer import ReTimer           
+
 import psutil
+from lxml import etree
 from logging.handlers import SysLogHandler  #
 
+BASE_PREFIX = ""
 
 class YTReader:
 	#parsing configuration xml to get video url
@@ -32,12 +41,12 @@ def __init__(self):
 
 	def update_values(self):
 		try:
-			video = pafy.new( URL )
+			video = pafy.new( self.URL )
 			ipc.set_read(0, video.viewcount)
 			ipc.set_read(1, video.likes)
 			ipc.set_read(2, video.dislikes)
 			ipc.set_read(3, int(10 * video.rating) )
-			video2 = pay.new( URL2 )
+			video2 = pafy.new( self.URL2 )
 			ipc.set_read(4, video2.viewcount)
 			ipc.set_read(5, video2.likes)
 			ipc.set_read(6, video2.dislikes)
@@ -70,15 +79,12 @@ def main(argv=None):
 		global ipc
 		sys.path.append('/opt/szarp/lib/python')
 		from test_ipc import TestIPC
-		ipc = TestIPC("example-base", "/opt/szarp/example-base/pdmn_youtube.py")
+		ipc = TestIPC("%s"%BASE_PREFIX, "/opt/szarp/%s/pdmn_youtube.py"%BASE_PREFIX)
 
 	ex = YTReader()
 	time.sleep(10)
 
 	ReTimer(3600, ex.update_values).go()
-	#while True:
-	#	ex.update_values()
-	#	time.sleep(3600)
 
 
 # end of main()