Installation

1. Prerequisites
2. Deploy services
3. Register aggregator and poller hosts
4. Test

This guide takes you through the steps required to install TSP on a cluster of RedHat machines. TSP is a core service just like crond or syslogd so you can expect all the usual niceties to be available: rpm-based deployment, man pages, canonical placement of configs and logs, automatic log rotation, and so on.

2.1. Prerequsites

To make it easy to set up a local mirror, TSP is released in the form of a yum repository (i.e. a set of rpms and the related yum metadata). Upload the repo to your local yum server, and add it to the repo list:

cat >/etc/yum.repos.d/opentsp.repo <<EOF
[opentsp]
name = Time Series Pipeline
baseurl = http://yum.example.com/opentsp
EOF

Prepare the following machines:

• controller - two or more lightweight machines.
• aggregator - one high-power machine (rule of thumb: 2 cores for every 2G of RAM).
• poller - one medium-power machine (rule of thumb: 1 core for every 1G of RAM).

(It is okay to use one powerful machine to run both aggregator and poller.)

Ensure that the following network configuration is in place:

• Load balancer TCP virtual IP for the controller service: controller-vip:80 maps to port 8084 on controller machines
• Firewall rule to allow TCP connectivity from 0.0.0.0/0 to controller-vip:80
• Firewall rule to allow TCP connectivity from 0.0.0.0/0 to aggregator-ip:4242

Installation of packages will consume a few megabytes of disk space. Thereafter, the only use of disk is logging of occasional diagnostic message.

2.2. Deploy services

2.2.1. controller

tsp-controller(8) is a service responsible for cluster management. All TSP processes follow instructions served by the controller.

Start it on every controller machine:

yum install tsp-controller
service tsp-controller start
chkconfig tsp-controller on

2.2.2. forwarder

tsp-forwarder(8) is a lightweight agent that collects metrics and forwards them to any subscribers.

Start it on every server in the data center:

yum install tsp
echo CONTROL_HOST=controller-vip:8084 >/etc/sysconfig/tsp
service tsp start
chkconfig tsp on

2.2.3. poller

tsp-poller(8) is a service that collects metrics from remote hosts. It is used in situations where the forwarder agent cannot be installed, for example in case of proprietary SNMP-only appliances or Internet APIs.

Start it on the poller machine:

yum install tsp-poller
echo CONTROL_HOST=controller-vip >/etc/sysconfig/tsp-poller
service tsp-poller start
chkconfig tsp-poller on

2.2.4. aggregator

tsp-aggregator(8) is a service that receives feeds from individual servers, and forwards it as a single stream to any subscribers.

Start it on the aggregator machine:

yum install tsp-aggregator
echo CONTROL_HOST=controller-vip >/etc/sysconfig/tsp-aggregator
service tsp-aggregator start
chkconfig tsp-aggregator on

2.3. Register aggregator and poller hosts

Forwarders must learn the address of the aggregator host before they can forward their feed to it.

Controller must learn the address of the poller host so that it can prevent overpolling. It's a protection against accidental startup of an extra poller instance. Without this protection, two pollers would poll the same host, producing duplicate metric data and potentially overloading the host.

Both steps are done by editing /etc/tsp-controller/network. Replace the dots with fully-qualified domain names:

<network>
    <aggregator host="..."/>
    <poller host="..."/>
</network>

Apply the new settings:

service tsp-controller restart

Controller will propagate updated configuration to the forwarders (and the poller). The propagation delay is about 1 minute.

2.4. Test

Check health of the pipeline by running the following command on the aggregator host:

tcpdump -Alnp -i any "dst net localnet and dst port 4242" | grep "put "

On average, it prints one line per each incoming data point. Expect to see mentions of every host that had forwarder installed. Look for a substring that looks like host=foo01.example.com. If data is missing, read the logs under /var/log/tsp.

Continue to Tutorials.

Return to Documentation.