Demo Stack

#Introduction

This page will walk you through the steps needed to launch an RTB stack. Of course, this is not meant to be a full production deployment of any kind but a presentation to showcase how the different components interact with each other.

One word before we begin: Depending on your setup, this could be spending real money. So, beware and use wisely.

#Preparation

First, make sure that you have a working build of RTBkit and that all tests are passing. The process is described here.

#Apache ZooKeeper

To be able to find each other, RTBkit components use a central naming service implemented with ZooKeeper. Therefore, you need a running instance of ZooKeeper. For this demo, we will use a very basic configuration for a local standalone server.

Simply run:

cp sample.zookeeper.conf ~/local/bin/zookeeper/conf/zoo.conf
~/local/bin/zookeeper/bin/zkServer.sh start

If you already have an instance running, you will be able to specify it later.

#Redis

Next on the list is our persistence backend. This is used by the banker to safely backup the tree of accounts that are needed for doing distributed accounting.

From the rtbkit folder, simply run:

redis-server ./sample.redis.conf

If you already have an instance running, you will be able to specify it later.

#Graphite

This step is optional but recommended. RTBkit components use Graphite to report various metrics and events that are happening within the system in real-time. Without a tool like this, the system is pretty opaque and it's really hard to tell what is going on. The first step is to install the required components:

pip install carbon
pip install whisper
pip install graphite-web

Now, to start the Carbon service that RTBkit components communicate with, you can use the example configuration files directly:

cd /opt/graphite/conf
for i in `ls *`; do sudo cp $i `echo $i | sed "s/\.example//g"`; done

and run :

/opt/graphite/bin/carbon-cache.py start

Now, to setup Graphite itself, follow the installation instructions on the web site and configure your web server accordingly.

#Mock Exchange

In RTBkit, we have added a simple mock exchange to validate the stack before going into production. We'll be using that to begin with and we will make sure that the whole thing works before pointing to a real exchange. The mock exchange simply generates a number of synthetic bid requests and randomly returns wins.

To start it, simply run:

./build/x86_64/bin/mock_exchange_runner

By default, this will send bid requests on port 12339 and wins notification on port 12340.

#RTBkit

With the preparation out of the way, let's focus on RTBkit components. To be able to run, we need:

• the request router to receive bid requests
• at least one agent that will perform the actual bidding
• the banker to manage accounts
• an optional augmenter to add content to a bid request
• some interface to an ad server to process wins
• the post auction loop service to process results from the ad server
• a data logger to store events on disk
• the agent configuration service
• and the monitor to track the sanity of the whole stack

That's quite a few processes. For this demo, you will use the fixed price bidding agent and the frequency cap augmenter that are available as examples in RTBkit.

To make things simpler, we have created a launcher application that starts those process along with a launch script.

Simply run:

./build/x86_64/bin/launcher --node localhost --script ./launch.sh sample.launch.json

This should start a tmux session that looks like the following:

By pressing Ctrl-B and 7, you can see the router processing bids. Right now, all bids are performed by the agent but they get rejected because there's no budget. Thus, with Ctrl-B and 8, you can't win any bids so nothing is happening in the post auction loop.

So let's add some budget.

The agent currently is tied to a budget account named hello:world. Since the banker has a REST API, you can take a look in your browser or by doing this:

curl http://localhost:9985/v1/accounts

To add budget, simply do a POST request like so:

curl http://localhost:9985/v1/accounts/hello/budget -d '{ "USD/1M": 123456789 }'

Note that this notation specify that the amount is in micro dollars. Thus, this is about 125$.

When budget gets added, you should start to see wins coming back to the exchange:

$ ./build/x86_64/bin/mock_exchange_runner
publishing on port 12339
publishing on port 12340
win sent payload={"account":["hello","world"],"adSpotId":"1","auctionId":"18238","bidTimestamp":0.0,"channels":null,"timestamp":1365793563.009584,"type":1,"uids":{"prov":"bar","xchg":"foo"},"winPrice":[1838083,"USD/1M"]}

win sent payload={"account":["hello","world"],"adSpotId":"1","auctionId":"18243","bidTimestamp":0.0,"channels":null,"timestamp":1365793563.034394,"type":1,"uids":{"prov":"bar","xchg":"foo"},"winPrice":[910538,"USD/1M"]}

win sent payload={"account":["hello","world"],"adSpotId":"2","auctionId":"18249","bidTimestamp":0.0,"channels":null,"timestamp":1365793563.062883,"type":1,"uids":{"prov":"bar","xchg":"foo"},"winPrice":[78770,"USD/1M"]}

win sent payload={"account":["hello","world"],"adSpotId":"2","auctionId":"18641","bidTimestamp":0.0,"channels":null,"timestamp":1365793564.020001,"type":1,"uids":{"prov":"bar","xchg":"foo"},"winPrice":[112679,"USD/1M"]}

win sent payload={"account":["hello","world"],"adSpotId":"1","auctionId":"18643","bidTimestamp":0.0,"channels":null,"timestamp":1365793564.031263,"type":1,"uids":{"prov":"bar","xchg":"foo"},"winPrice":[1029505,"USD/1M"]}

win sent payload={"account":["hello","world"],"adSpotId":"1","auctionId":"18652","bidTimestamp":0.0,"channels":null,"timestamp":1365793564.086975,"type":1,"uids":{"prov":"bar","xchg":"foo"},"winPrice":[396665,"USD/1M"]}

To stop the whole stack, simply hit Ctrl-C on pane 0 to close it. Note that all the other panes are simply showing logs. Those can be found in ./logs

#Notes

Support for real exchanges is currently being added. Stay tuned.