(BITS-VIB) geek-tools

These scripts are meant to help advanced users better use CLI programs and build workflows for their research.

Application monitoring

Bash and R plotting scripts to monitor a demanding application and plot the results to 'png' or 'pdf'. Working with BIG data often requires to estimate and optimize CPU and or memory usage. This pair of tools records a running application to evaluate its needs and create a plot to visualize the results.

Creating a monitoring file with logmyapp

Start your favorite CLI program (or scrip running it) and monitor cpu and ram with logmyapp

REM: The bash script is only functional under Unix, the Apple Darwin top command lacking the -b option does not allow running it (if you know how to fix this, please let me know!!).

Usage: logmyapp [name of the app to monitor] [interval (sec|default=5)]

Plotting from the log file with log2plot.R

Then use the logmyapp text log file is used as input by log2plot.R.

In order to use this script, you will need [R] and RScript installed on your computer (done by most package installers including yum and apt-get). You will also need the [R] package optparse http://cran.r-project.org/web/packages/optparse/ to handle command line arguments.

Installing the dependencies is documented on the top of the code. Please read the respective package documentations if you wish to improve these scripts.

log2plot.R -h

Usage: ./log2plot.R [options]

Options:
	-i INFILE, --infile=INFILE
		input file name

	-r TOTRAM, --totram=TOTRAM
		total RAM available on the computer [default: 4]

	-m MAXRAM, --maxram=MAXRAM
		max RAM value on y-scale [default: totram]

	-t TITLE, --title=TITLE
		Graph Main Title

	-o OUTFILE, --outfile=OUTFILE
		base name for output [default: monitoring_plot]

	-f OUTFORMAT, --outformat=OUTFORMAT
		file format for output 1:PNG, 2:PDF [default: 1]

	-h, --help
		Show this help message and exit

Example screenshots

The output of a 15 million paired end reads bowtie2 mapping job to the human reference genome (hg19) is shown as example (bowtie2_usage1401561413.log):

#time pid %cpu %MEM App
11 1223 7.8 0.2 bowtie2-align-s
16 1223 7.8 2.8 bowtie2-align-s
22 1223 7.8 5.4 bowtie2-align-s
27 1223 9.8 8.0 bowtie2-align-s

... many more lines here

980 1223 685.0 22.6 bowtie2-align-s
985 1223 756.6 22.6 bowtie2-align-s
991 1223 780.5 22.6 bowtie2-align-s

The monitoring data was fed to the accompanying R script with the following command:

log2plot.R -i bowtie2_usage1401561413.log \
    -r 16 \
    -m 6 \
    -f 1 \
    -t "bowtie2 monitoring" \
    -o bowtie2_monitoring

The resulting picture reports bowtie2 activity and RAM usage (here in png format)

logging all outputs to a file for inspection with runandlog

Uses tee to copy all output, including standard error messages to a log file. After completion, the script reports the time spent by the command.

Log temperature in the server room for plotting or simply controlling

A basic Perl script logtemp.pl is used to monitor the Temper-1F measures to file. A simple example of gnuplot code is added to generate images. This is very basic and heavily relies on the python package temper-python used to master the cheap USB device.

The customized bash/gnupot script logtemp2plot.sh takes care of making a nice image from the log results.

logtemp2plot.sh -h

 Usage: logtemp2plot.sh <logtemp.pl output file>
 # script version 1.0, 2016_08_05
 #  -i <inputfile|templog.txt>
 ## optional parameters (|default value)
 #  -s <from time|0>
 #  -e <until time|default to last recorded>
 #  -b <min scale temp|0>
 #  -t <max scale temp|40>

Example

The output of ambiant temperature logging in our computer server room is shown as example (templog.txt).

Feeding the text file to the gnuplot tool with the command:

logtemp2plot.sh -i templog.txt -s 80000 -e 110000 -b 18 -t 24

produced the following picture

---

come back later for more

Please send comments and feedback to bits@vib.be

------------

This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.