Getting Started

This guide will show you how to set up a machine and user environment to build RTBkit and run a demo program.

There are three levels of setup:

1. you need need to have a machine with all the right packages installed
2. you need to install some packages within a user's home directory
3. you need to build RTBkit.

1) Machine-level Setup

Important note: if you are using Amazon instances for this, please do not use a Micro instance, as it will have insufficient RAM to compile. Amazon Small instances should be sufficient.

This step and the next one (user-level setup) can be skipped by launching an Amazon instance based on ami-7b531a12 available in the US East (N. Virginia) region: the right packages are installed and the user-level setup has already been done for the rtbkit user.

RTBkit was developed and is tested to run on Ubuntu 12.04.2 LTS (Precise Pangolin) 64-bit and to be compiled with gcc 4.6. Note that a 64-bit system is currently required because of the 4GB memory limit of 32-bit systems.

Make sure you have at least 16GB of disk for building from the source. Also, your CPU should have support for SSE4.2 as it greatly improves performances.

You should run the following command to ensure you have all of the required dependencies which can be installed via apt:

$ sudo apt-get install git-core g++ libbz2-dev \
liblzma-dev libcrypto++-dev libpqxx3-dev scons libicu-dev \
strace emacs ccache make gdb time automake libtool autoconf \
bash-completion google-perftools libgoogle-perftools-dev \
valgrind libACE-dev gfortran linux-tools uuid-dev liblapack-dev \
libblas-dev libevent-dev flex bison pkg-config python-dev \
python-numpy python-numpy-dev python-matplotlib libcppunit-dev \
python-setuptools ant openjdk-7-jdk libcppunit-dev doxygen \
libfreetype6-dev libpng-dev python-tk tk-dev python-virtualenv \
liblzma-dev sshfs rake ipmitool mm-common libsigc++-2.0-dev \
libcairo2-dev libcairomm-1.0-dev

Followed by this command to remove system packages that are known to cause problems with platform-deps:

$ sudo apt-get purge libcurl4-openssl-dev

2) User-level Setup

This setup needs to be done for the user which will be used to compile/run RTBkit. You should never use the root user to run RTBkit and therefore, should do the whole setup with the user you intend to run with.

Environment Variables

Add the following to your ~/.profile and/or whatever your shell requires for the equivalent effect:

# Add local directory for libraries, etc
mkdir -p $HOME/local/bin
PATH="$HOME/local/bin:$PATH"
 
mkdir -p $HOME/local/lib
export LD_LIBRARY_PATH="$HOME/local/lib:$LD_LIBRARY_PATH"
 
export PKG_CONFIG_PATH="$HOME/local/lib/pkgconfig/:$HOME/local/lib/pkg-config/"

To see if your environment is properly setup, make sure you get an output similar to this:

$ env | grep PATH
LD_LIBRARY_PATH=/home/rtbkit/local/lib:
PATH=/home/rtbkit/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
PKG_CONFIG_PATH=/home/rtbkit/local/lib/pkgconfig/:/home/rtbkit/local/lib/pkg-config/

platform-deps

You'll need to install a bunch of dependencies which are wrapped up in the Datacratic platform-deps repo:

$ git clone https://github.com/datacratic/platform-deps.git
$ cd platform-deps
$ git submodule update --init
$ make all

If you have trouble compiling these dependencies, you may find some help in the platform-deps README. One known issue is that it takes a fair bit of memory to compile these dependencies depending on how much parallelization you use with make. Another known issue is the use of sse4.2 instructions when building cityhash which can sometimes result in illegal instructions being raised when used on migrated VMs.

Note that in an initial build and you should not use the -jX option. Even if this takes quite a bit of time, the different submodules need to be compiled in sequence.

3) Building RTBkit

Now you're ready to build RTBkit proper, logged in as the user for which the above "user-level setup" has been performed. If you are using the pre-built AMI mentioned at the top of this page, the user is rtbkit.

$ sudo su - rtbkit
$ git clone https://github.com/rtbkit/rtbkit.git
$ cd rtbkit
$ git submodule update --init
$ cp jml-build/sample.local.mk local.mk
$ make dependencies
$ make compile
$ make test

For the test suite to pass it is currently required that you do not have a bootstrap.json in your project root and that you do not try to run make test in parallel mode (-jX option).

Now what?

The best entry-points into the RTBkit source are in the /examples directory, especially the rtbkit_integration_test, which can be run with make rtbkit_integration_test

Also, some of those example components are used by the Demo Stack that provides a simple runnable RTB stack using a synthetic exchange.