NNTSC 2.25 --------------------------------------------------------------------------- Copyright (c) 2016, 2017 The University of Waikato, Hamilton, New Zealand. All rights reserved. This code has been developed by the University of Waikato WAND research group. For further information please see http://www.wand.net.nz/. --------------------------------------------------------------------------- NNTSC is a system designed for collecting, storing and examining network time series data. The main feature of NNTSC is that it is flexible and can work with just about time series data, so you can unify your measurement storage and analysis systems around a single data management API. NNTSC also provides a client API that allows other programs to connect to the measurement database and either ask for historical data or register interest in upcoming live data. If you have any questions, comments or suggestions about NNTSC, please email us at email@example.com. Quickstart Instructions for Debian Users ======================================== 1. Install the NNTSC Debian package 2. Create a Postgresql database for NNTSC: createdb nntsc 3. Create an Influx database for NNTSC: influx -execute 'CREATE DATABASE nntsc' 4. Configure NNTSC (see 'Configuration' section below for more details) 4.1. Create an RRD List file (if needed). 4.2. Edit /etc/nntsc/nntsc.conf to point to your new database and RRD list. 5. Run: build_nntsc_db -C /etc/nntsc/nntsc.conf 6. Run: /etc/init.d/nntsc start Installation from Source ======================== NNTSC uses Python setuptools to install both itself and its dependencies. In version 2.3 there has been a significant change to the installation process -- some of the NNTSC dependencies are now distributed and installed separately, namely libnntscclient. If you have checked out a copy of NNTSC from our git repository, you will also need to get libnntscclient from the repository as well. Install it by changing into the directory and running 'python setup.py install' prior to attempting to install NNTSC itself. Once these are installed, simply run the following to install NNTSC: python setup.py install By default, NNTSC installs into your system's dist-packages directory but you will need root privileges to install it there. You can change the installation location using the --prefix option. Run 'python setup.py --help install' for more details about how you can customise the install process. Supported Network Measurement Data Sources ========================================== At present, NNTSC only supports a handful of network measurement methods, but we are working to extend this as time goes on. NNTSC has support for the following data sources: AMP -- the Active Measurement Project Smokeping RRDs Glossary ======== Here are a few important terms that we use when talking about NNTSC. collection: A collection is a group of network measurement time series' (which we call streams) which use the same measurement approach and result structure. For example, each AMP test would result in a separate collection. dataparser: The Python code that processes incoming network measurements and inserts them into the NNTSC database. Each collection has its own dataparser. metric: A term that describes what the data value for a time series actually represents. For instance, the ping values stored by the NNTSC smokeping collection and the rtt values stored by the AMP ICMP collection both measure latency, so the metric for these is 'latency'. A traceroute path length is measured in hops, so the metric in this case is 'hops'. module: Refers to the input source for a collection. The name 'module' is used because many collections share the same basic input source, e.g. all the AMP test collections deal with the AMP message broker, so the dataparsers for those collections all end up using the same Python module for receiving and processing messages from the shared source. modsubtype: Used to differentiate between collections that share the same input source. Short for 'module subtype', where the subtype can be used to determine which dataparser should be used to process and store the received measurements correctly. resolution: When discussing RRDs, resolution refers to the frequency at which measurements are taken. NNTSC only stores measurements taken at the highest available resolution and will not store RRD data that has been aggregated. stream: A stream is a single network measurement time series within NNTSC. Each stream has a unique ID number and has a series of properties that describe how the measurements were taken. Configuration ============= Before you start using NNTSC, you'll need to create a config file that defines where your time series data is going to be coming from and where you want to store it. An example configuration file is included in conf/nntsc.conf which you can use as a starting point. If you've installed NNTSC from a Debian package, this same file will be installed to /etc/nntsc/nntsc.conf and you should edit that file to add your configuration. The configuration options use the standard python ConfigParser format, where options are grouped into sections. The sections currently supported by NNTSC and their options are as follows: [database] Options relating to the Postgresql database where the relational data will be stored. Available options: host - the host where the database is located. Leave blank if the database is on the local machine. database - the name of the NNTSC database. username - the username to use when connecting to the database. Leave blank to use the name of the user running NNTSC. password - the password to use when connecting to the database. Leave blank if no password required. [influx] Options relating to the Influx database where the time series data will (mostly) be stored. Available options: useinflux - if 'no', all time series data will be stored in the Postgresql database instead. database - the name of the Influx database. username - the username to use when connecting to the database. Leave blank to use the name of the user running NNTSC. password - the password to use when connecting to the database. Leave blank if no password required. host - the host where the database is located. Leave blank if the database is on the local machine. port - the port that the Influx daemon is listening on. Use 8086 unless you've configured Influx to use another port. keepdata - the length of time to keep data in the database. For example, '30d' will keep data for 30 days. [liveexport] Options relating to the Rabbit queue that funnels newly collected data to the live exporter. Available options: queueid - the name of the message queue. port - the port that the local Rabbit instance is listening on. username - the username to use when connecting to Rabbit. password - the password to use when connecting to Rabbit. [modules] These options are used to enable or disable NNTSC dataparsers. To enable a module, set the appropriate option to "yes". To disable a module, set the option to "no". Available options: amp - controls whether NNTSC accepts data from AMP. rrd - controls whether NNTSC should scrape data from RRDs. [amp] Options relating to connecting to an AMP message queue and receiving AMP data Available options: username - the username to use when connecting to AMP's message queue. password - the password to use when connecting to AMP's message queue. host - the host where the AMP message broker is running. port - the port to connect to on the AMP message broker host. ssl - if True, encrypt all messages using SSL. If False, use plain-text. queue - the name of the message queue to read AMP data from. commitfreq - the number of AMP messages to process before committing. [rrd] Options relating to scraping data from existing RRD files. See "RRD Scraping" below for more details on how to configure RRD scraping. Available options: rrdlist - the full path to a file describing the RRDs file to scrape. RRD Scraping ============ NNTSC includes the ability to harvest measurements from the RRD files created by other measurement tools. This allows users to deploy NNTSC alongside their existing monitoring setup. At present, NNTSC only supports scraping the RRD files produced by Smokeping. In the Smokeping case, the Smokeping tool runs as per usual, writing its results to an RRD file. NNTSC will then read the most recent measurements from the RRD file and store them in its own database for later analysis. Unlike the RRD, though, NNTSC will keep the measurements in their original format for as long as they remain in the database, rather than aggregating them once they reach a certain age. The RRD list file is used to tell NNTSC which RRD files it should read and what is being measured by each file. A documented example RRD list file is provided in conf/rrd.examples which should be sufficient to help you write your own RRD list file. When you run the script to build your NNTSC database, the RRD list file will be parsed and appropriate streams will be created for each RRD that is described in the file. Once NNTSC itself is running, the specified RRDs will be periodically checked for new datapoints. Should you decide you add new RRDs to your RRD list, you must re-run the database build script for NNTSC to recognise them. Building the Database ===================== Before you can run NNTSC itself, you will need to create and initialise the database that will be used to store the time series data that you are collecting. NNTSC uses a postgresql database for its storage. First, decide on a name for your database -- in the following examples, I am going to use 'nntsc' as the database name but you can choose anything you like. Next, create a Postgresql database and ensure that it is owned by the user that you will be ultimately connecting to the database as. In the simplest case, I will be connecting using my own username so I can create the nntsc database using: createdb nntsc If you're using Influx to store the time series data, you'll also need to create an Influx database. Easiest way to do this via the InfluxQL shell. influx -execute 'CREATE DATABASE nntsc' The next step is run the build_nntsc_db script that should have been installed on your system when you installed NNTSC. This script will create all of the required tables for storing the time series data. In the case of some input sources, e.g. RRDs, it will also create 'streams' for each of time series that will be collected. Make sure the database section of your config file now refers to your new database and then simply run the script as follows: build_nntsc_db -C <your config file> If you update your configuration, e.g. you wish to enable a data source that you had earlier disabled or you have added new RRDs that you want to collect data for, you can re-run build_nntsc_db and it will add new tables and streams as needed without overwriting any existing ones. If you really do want to start over fresh, you can pass a -F flag to build_nntsc_db. Be warned, this will drop ALL of your existing tables and any time series data and streams that are stored within them. Don't do this unless you are really sure you want to lose all that data! If you explore your new Postgresql database with the psql tool, you should have a 'collections' table and a number of tables begining with 'stream_' and 'data_', one per collection. Starting NNTSC ============================ Once you've created and initialised your database, you can start NNTSC in several ways. Firstly, if you have installed NNTSC via a Debian package, there will be an init script installed in /etc/init.d/ that you can use to start and stop a NNTSC daemon. This is by far the best available option. Using the script, you can start NNTSC by running the following: /etc/init.d/nntsc start The init script also accepts 'stop' and 'restart' as commands, so it can be used to easily stop a running instance of NNTSC. The init script will use /etc/nntsc/nntsc.conf as the NNTSC configuration file so either make sure that file contains your desired config or edit the init script to point to your own config file. Alternatively, you may manually start an NNTSC daemon by running the following: nntsc -b -C <your config file> [-p port] [-P pidfile] Finally, you may run NNTSC in the foreground by omitting the '-b' option from the manual command given above. In this case, we strongly recommend running NNTSC within screen or some other persistant terminal environment. By default, the nntsc exporter will listen on port 61234 for incoming client connections. This can be changed using the '-p' option. Note that in all cases, the same config file must be used by both NNTSC and the build_nntsc_db script. If run as a daemon, NNTSC will write any error or log messages to /tmp/nntsc.log, otherwise it will report them directly to standard error. Querying the NNTSC Database =========================== While running, nntsc also listens on TCP port 61234 for client connections. Connected clients may query NNTSC to access any data currently stored in the time series database or register an interest in live data as it arrives from the various input sources. There is a Python API for writing a client that can connect to and query NNTSC, without the programmer needing to know the underlying communication protocol. The code for this API is located in the libnntscclient repository -- it's not very well-documented at the moment but hopefully we can improve that in future releases. To use the client API, just add the following imports to your Python program: from libnntscclient.protocol import * from libnntscclient.nntscclient import NNTSCClient There are three types of messages that a client can send to the NNTSC collector: NNTSC_REQUEST -- requests information about the current collections or streams NNTSC_SUBSCRIBE -- requests measurement data for a set of streams NNTSC_AGGREGATE -- requests aggregated measurement data for a set of streams NNTSC_REQUEST messages must also specify a request type: NNTSC_REQ_COLLECTION -- request a list of all available collections NNTSC_REQ_STREAMS -- request a list of streams for a given collection NNTSC_REQ_SCHEMA -- request the list of columns in the stream and data tables for a given collection NNTSC_SUBSCRIBE messages can be used to get all of the available measurements for some set of streams over a given time period. If the time period includes historical data, any available historical data will be immediately returned in response. If the time period ends in the future, new measurements will be exported to the client as they arrive until the end time is passed. If the end time is set to 0, then live data will be exported until the client disconnects. NNTSC_AGGREGATE messages work much the same as NNTSC_SUBSCRIBE messages, except that the measurements are aggregated into bins of a given size. NNTSC_AGGREGATE can only be used with historical data -- if the end time is in the future, then you will only get data up to the current time. You must also provide an aggregator function and a list of columns from the data table that you want to apply the aggregation to. Supported aggregation functions are: max, min, sum, avg and count. They should be fairly self-explanatory as to how they will aggregate the data within a bin.