Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

The ServerUsage system is designed to collect and process statistic information from computers running a GNU-Linux Operating System.

branch: master
README.md

ServerUsage - README

  • Name: ServerUsage

  • Version: 6.3.8

  • Release date: 2012-11-29

  • Author: Nicola Asuni

  • Copyright (2012-2012):

Fubra Limited
Manor Coach House
Church Hill
Aldershot
Hampshire
GU12 4RQ
http://www.fubra.com
support@fubra.com

SOFTWARE LICENSE:

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see http://www.gnu.org/licenses/.

See LICENSE.TXT file for more information.

DESCRIPTION:

The ServerUsage system is designed to collect and process statistic information from computers running a GNU-Linux Operating System.

This project is composed by:

ServerUsage-Server

The ServerUsage-Server program listen on a TCP port for incoming log data (from ServerUsage-Client) and store them on a SQLite table. An script (serverusage_dbagg.sh) is executed periodically to aggregate data on another table and delete obsolete data.

The serverusage_api.php script can be remotely used to extract formatted data from the database or display graphs.

ServerUsage-Client

The ServerUsage-Client program is composed by two main sections: the SystemTap (http://sourceware.org/systemtap) serverusage_client.ko kernel module to collect and output usage statistics of the machine where it is installed, and the serverusage_tcpsender.bin to send the output to a log server via TCP.

ServerUsage-Client-MDB

The ServerUsage-Client-MDB program is used to collect user statistics form a MariaDB database.

GENERAL USAGE SCHEMA

(ServerUsage-Client) - - TCP CONNECTION - -> (ServerUsage-Server)

([serverusage_client.ko]==>[serverusage_tcpsender.bin]) - - TCP CONNECTION - -> ([serverusage_tcpreceiver]==>[SQLite log_raw table (raw data)])

[SQLite log_raw table]==>[serverusage_dbagg.sh]==>[SQLite log_agg_hst table (aggregated data)]

[SQLite log_agg_hst table]==>[serverusage_api.php]==>[Data with selected format or SVG graph]

API

The serverusage_api.php script can be remotely used to extract formatted data from the database or display graphs.

PARAMETERS:

from: (integer) starting timestamp in seconds since EPOCH;
to: (integer) starting timestamp in seconds since EPOCH - this value can be at maximum equal to (current_time - (2 * DB_AGGREGATION_DELAY)) that is also the default value;
metric: (not available with svg mode) type of info to extract; Possible values are: 'uid', 'ip', 'uip', 'grp', 'glb', 'all'. The return values for each metric are:
    uid : user_id, cpu_ticks;
    uidt : user_id, cpu_ticks, minimum start time, maximum end time;
    ip  : ip, net_in, net_out;
    ipt  : ip, net_in, net_out, minimum start time, maximum end time;
    uip : user_id, ip, cpu_ticks, io_read, io_write, net_in, net_out;
    uipt : user_id, ip, cpu_ticks, io_read, io_write, net_in, net_out, minimum start time, maximum end time;
    grp : start_time, end_time, user_id, ip, cpu_ticks, io_read, io_write, net_in, net_out;
    glb : (default for SVG mode) lah_start_time, lah_end_time, lah_cpu_ticks, lah_io_read, lah_io_write, lah_netin, lah_netout;
    all : start_time, end_time, process, user_id, ip, cpu_ticks, io_read, io_write, net_in, net_out.
uid : (integer) if set, filter result for the requested user ID;
ip : (IP address) if set, filter result for the requested IP address.
mode: output format ('json' = JSON, 'csv' = CSV TAB-Separated Values, 'psa' = base64 encoded PHP Serialized array, 'svg' = SVG).

ADDITIONAL PARAMETERS FOR SVG MODE:

width: (integer) optional width for SVG output (default 1024; minimum 50).
height: (integer) optional height for SVG output; will be rounded to a multiple of 5 (default 750, minimum 50).
scale: linear = vertical linear scale (default), log = vertical logarithmic scale.
bgcol: type of background color: 'dark' or 'light' (default). Note: the SVG background is always transparent.
gtype: sequence of number representing the graphs to display: 1 = CPU TICKS, 2 = IO READ, 3 = IO WRITE, 4 = NET IN, 5 = NET OUT. Default: 12345.

USAGE EXAMPLES:

JSON:
    serverusage_api.php?from=1332769800&to=1332845100&metric=uid&mode=json
    serverusage_api.php?from=1332769800&metric=uidt&mode=json
    serverusage_api.php?from=1332769800&to=1332845100&metric=ip&mode=json
    serverusage_api.php?from=1332769800&to=1332845100&metric=ipt&mode=json
    serverusage_api.php?from=1332769800&to=1332845100&metric=uip&mode=json
    serverusage_api.php?from=1332769800&to=1332845100&metric=uipt&mode=json
    serverusage_api.php?from=1332769800&to=1332845100&metric=all&mode=json
    serverusage_api.php?from=1332769800&to=1332845100&metric=all&uid=320&mode=json

CSV:
    serverusage_api.php?from=1332769800&to=1332845100&metric=all&uid=320&mode=csv

BASE64 ENCODED PHP SERIALIZED ARRAY:
    serverusage_api.php?from=1332769800&to=1332845100&metric=all&uid=320&mode=psa

SVG:
    serverusage_api.php?from=1332769800&to=1332845100&mode=svg&width=1024&height=750&scale=log
    serverusage_api.php?from=1333532663&to=1333627917&mode=svg&scale=log&bgcol=light&gtype=12345
    serverusage_api.php?from=1333532663&to=1333627917&mode=svg&scale=linear&bgcol=light&gtype=15
    serverusage_api.php?from=1333532663&to=1333627917&mode=svg&scale=log&bgcol=light&gtype=5

OUTPUT:

The output format can be JSON (JavaScript Object Notation), CSV (tab-separated text values), Base64 encoded serialized array or SVG (Scalable Vector Graphics).

ADDITIONAL NOTES:

The reference time for all servers should be the standard UTC.
The latest available time on the ServerUsage-Server aggregated table is always in the past by the value specified by DB_AGGREGATION_DELAY constant (by default 5 minutes).
A valid time interval, for example, can be calculated as follows:

    polling_interval = 900; // 15 minutes * 60 seconds; must be equal or greater than DB_AGGREGATION_DELAY.
    delay_time = 600; // 10 minutes * 60 seconds; must be equal or greater than (2 * DB_AGGREGATION_DELAY).
    end_time = (current_time - delay_time);
    start_time = (end_time - polling_interval);

HOW-TO CREATE ServerUsage RPMs

This is a short hands-on tutorial on creating RPM files for the ServerUsage project. For an automatic building script for CentOS and the latest RPM packages please check the CatN Repository: https://github.com/fubralimited/CatN-Repo

NOTE: The sever configuration for ServerUsage-Client and ServerUsage-Server may be different, so this process must be executed in different environments.

DEVELOPMENT ENVIRONMENT

To build RPMs we need a set of development tools. This is a one-time-only setup, installed by running those commands from a system administration (root) account. NOTE: You may need to change the the

Install the EPEL repository:

# rpm -Uvh http://download.fedoraproject.org/pub/epel/6/$(uname -m)/epel-release-6-7.noarch.rpm

Install development tools and Fedora packager:

# yum install @development-tools
# yum install fedora-packager

The following packages are required to create ServerUsage RPMs:

# yum install kernel-devel elfutils-devel sqlite-devel.x86_64 sqlite.x86_64 MariaDB-devel

Install debug packages (change repository if you are not using CentOS):

wget http://debuginfo.centos.org/6/$(uname -m)/kernel-debug-debuginfo-$(uname -r).rpm
wget http://debuginfo.centos.org/6/$(uname -m)/kernel-debuginfo-$(uname -r).rpm
wget http://debuginfo.centos.org/6/$(uname -m)/kernel-debuginfo-common-$(uname -m)-$(uname -r).rpm
rpm -U --force kernel-debug-debuginfo-$(uname -r).rpm kernel-debuginfo-$(uname -r).rpm kernel-debuginfo-common-$(uname -m)-$(uname -r).rpm

Install SystemTap:

# yum -y install systemtap systemtap-client systemtap-devel systemtap-runtime systemtap-initscript systemtap-grapher systemtap-sdt-devel systemtap-server systemtap-testsuite

Create a dummy user specifically for creating RPM packages:

# /usr/sbin/useradd makerpm
# passwd makerpm

Reboot the machine, log as makerpm user and create the required directory structure in your home directory by executing:

$ rpmdev-setuptree

The rpmdev-setuptree program will create the ~/rpmbuild directory and a set of subdirectories (e.g. SPECS and BUILD), which you will use for creating your packages. The ~/.rpmmacros file is also created, which can be used for setting various options.

CREATE THE ServerUsage RPMs

Download the ServerUsage sources:

$ cd ~
$ git clone git://github.com/fubralimited/ServerUsage.git

Copy the SPEC files and source files to rpmbuild dir:

$ cd ~/ServerUsage
$ export SUVER=$(cat VERSION) 

$ cd ~/ServerUsage/server
$ cp serverusage_server.spec ~/rpmbuild/SPECS/
$ tar -zcvf ~/rpmbuild/SOURCES/serverusage_server-$SUVER.tar.gz  *  

$ cd ~/ServerUsage/client
$ cp serverusage_client.spec ~/rpmbuild/SPECS/
$ tar -zcvf ~/rpmbuild/SOURCES/serverusage_client-$SUVER.tar.gz  *  

$ cd ~/ServerUsage/client_mdb
$ cp serverusage_client_mdb.spec ~/rpmbuild/SPECS/
$ tar -zcvf ~/rpmbuild/SOURCES/serverusage_client_mdb-$SUVER.tar.gz  *    

Create the RPMs:

$ cd ~/rpmbuild/SPECS/
$ rpmbuild -ba serverusage_server.spec  
$ rpmbuild -ba serverusage_client.spec  
$ rpmbuild -ba serverusage_client_mdb.spec  

The RPMs are now located at ~/rpmbuild/RPMS/$(uname -m)

Install ServerUsage-Server

The ServerUsage-Server RPM must be installed only on the Log Server (the computer receiving the logs from the clients).

As root install the ServerUsage-Server RPM file:

# rpm -i serverusage_server-6.3.8-1.el6.$(uname -m).rpm

Once the RPM is installed you can configure the ServerUsage-Server editing the following file:

# nano /etc/serverusage_server.conf

The ServerUsage-Server includes a SysV init script to start/stop/restart the service:

# /etc/init.d/serverusage_server start|stop|status|restart|reload|condrestart

The init script starts the serverusage_tcpreceiver.bin program that listen for incoming TCP connections from the clients, and install a cron job to aggregate the data every 5 minutes. The raw data received from serverusage_tcpreceiver.bin is stored on a SQLite 3 database (var/lib/serverusage/serverusage.db) table named log_raw. The table containing the aggregated data is called log_agg_hst. The aggregated data is immediately removed from the log_raw table. The data on log_agg_hst older than DB_GARBAGE_TIME seconds is automatically removed.

To start the service at boot you can use the following command:

# chkconfig serverusage_server on

To extract formatted information from the SQLite database you can use the serverusage_api.php. This file is installed by default in /var/www/serverusage directory, so you have to configure Apache/PHP accordingly or move the script to another position. The serverusage_api.php allows you to extract filtered information in various formats: JSON (JavaScript Object Notation), CSV (tab-separated text values), Base64 encoded serialized array or SVG (Scalable Vector Graphics). On the same directory where the file serverusage_api.php is, you can find an example HTML file that displays an auto-update graph using the php API.

Install ServerUsage-Client

The ServerUsage-Client RPM must be installed on each client computer to monitor.

As root install the SystemTap-Runtime and ServerUsage-Client RPM files:

# yum -y install systemtap-runtime
# rpm -i serverusage_client-6.3.8-1.el6.$(uname -m).rpm

Configure the ServerUsage-Client

# nano /etc/serverusage_client.conf 

Set the IP address of the Log server where ServerUsage-Server is installed and be sure that the specified TCP port is open on both client and server.

The ServerUsage-Client includes a SysV init script to start/stop/restart the service:

# /etc/init.d/serverusage_client start|stop|status|restart|reload|condrestart

When the service is started, the serverusage_client.ko SystemTap kernel module is executed via the staprun command and the output is piped to the serverusage_tcpsender.bin to be sent to the Log server via a TCP connection. If the connection is broken or the Log server is not responding, the log files are temporarily stored on /var/log/serverusage_cache.log file and resent as soon the TCP connection is restored.

To start the service at boot you can use the following command:

# chkconfig serverusage_client on

Install ServerUsage-Client-MDB

The ServerUsage-Client RPM must be installed on the computer containing the MariaDB database.

As root install the ServerUsage-Client RPM file:

# rpm -i serverusage_client_mdb-6.3.8-1.el6.$(uname -m).rpm

Configure the ServerUsage-Client

# nano /etc/serverusage_client_mdb.conf 

Set the IP address of the Log server where ServerUsage-Server is installed and be sure that the specified TCP port is open on both client and server.

The ServerUsage-Client includes a SysV init script to start/stop/restart the service:

# /etc/init.d/serverusage_client_mdb start|stop|status|restart|reload|condrestart

To start the service at boot you can use the following command:

# chkconfig serverusage_client_mdb on
Something went wrong with that request. Please try again.