Redis Augmentor

The following page introduces the RTBkit Redis Augmentor, and how to use it.

Introduction

As described in this page, the Augmenter:

• Receives a series of bid requests and will add information to them in the form of tags or data. This information can be either per account or global to all accounts.
• Subscribes to the Agent Configuration Service to know the augmentation for each client(bidding agent)

The complete source code for the example is available in the rtbkit repository under the following 3 files:

• rtbkit/plugins/augmentor/redis_augmentor.h
• rtbkit/plugins/augmentor/redis_augmentor.cc
• rtbkit/plugins/testing/redis_augmentor_test.cc

This page describes how to write an Augmentor as an instance of the SyncAugmentor. We shall also assume that [accounts](https://github.com/rtbkit/rtbkit/blob/master/common/account_key. h) follow the <campaign>.<strategy> pattern. For more information on accounts, see the banker's documentation.

Architecture

RTBkit's Redis Augmentor is an AsyncAugmentor, which overrides its

void
RedisAugmentor::
onRequest(const AugmentationRequest & request, SendResponseCB sendResponse) /* override */;

, is composed of a:

RTBKIT::AgentConfigurationListener agent_config_;

and aggregates or compose of a

std::shared_ptr<Redis::AsyncConnection> redis_ ;

Given our RTBKIT::AugmentationList, a SendResponseCB is defined as following:

typedef std::function<void (const AugmentationList &)> SendResponseCB;

Redis::AsyncConnection is a thin layer around hiredis, which as far as we are concerned, provides an asynchronous, object oriented API around a physical connection to a Redis instance, with convenient timeout managements, as well as commands pipelining (which are both used in our case). Redis::AsyncConnection as it's own even loop, and on which augmentation callbacks will be called.

It is also important to realise that the inherited AsyncAugmentor, is composed (amongst other things) of a thread pool, which it will use to service inbound augmentation requests. The size of our servicing thread poll is a parameter to our base constructor. Our `Redis

It is important to understand the life cycle of a bid request; that is, as far as the augmentation process is concerned:

1. The exchange sends a bid request to the router where it is parsed by the exchange connector
2. The router relays the bid request to the Augmenter
3. The augmenter returns augmentations("+") to the router
4. The router sends augmented bid request("BR+") to the bidding agent

In the case of our RedisAugmentor, the sequence between 2 and 3 above, can be further specified:

1. The router relays the bid request to the RedisAugmentor
2. For each agent associated with this bid request, the RedisAugmentor tries to fetch an <augmentations>.<redis>.<config>.<aug-list> JSON::ArrayValue from each associated AgentConfigEntry, eventuelly upserting an ordered map jobs indexed by Redis keys, mapping to a set of AccountKey (<campaign>.<strategy>).
3. Out of the keys indexing jobs, the RedisAugmentor then builds a batch of Redis commands, and submits it as a batch to it's Redis::AsyncConnection member, along with a redis call defined as a lambda capturing our job map by value as well as the SendResponseCB passed as a parameter to the call. Note that this pipelined redis command are submited with a 4 milliseconds timeout.
4. Upon being called by by our on our redis thread, our call back submitted above, then constructs an AugmentationList according to both job and the Redis::Results passed as a callback argument by our Redis::AsyncConnection and calls SendResponseCB to pass it back to the calling AugmentorLoop run by our router.

Voilà.

Syntax of the keys fetched from the AugmentationRequest

The general structure of keys which our RedisAugmentor loooks for, is

 prefix:path:value

where prefix to be the arbitrary string "RTBkit:aug", and value is the value of the attribute of a bid request json object, as described by a path using the following syntax:

 - "." => root node
 - ".[n]" => elements at index 'n' of root node (an array value)
 - ".name" => member named 'name' of root node (an object value)
 - ".name1.name2.name3"
 - ".[0][1][2].name1[3]"
 - ".%" => member name is provided as parameter
 - ".[%]" => index is provied as parameter

In our case, the root node needs not to be provided in the key, as it is obviously the bid request in question. For a given agent, the keys are fetched from its associated AugmentationConfig configuration object, as path into the bid request passed along with the AugmentationRequest.

Example

(from rtbkit/plugins/testing/redis_augmentor_test.cc)

The following bid request:

 {"id":"85885bb0-b91b-11e2-c4cf-7fba90171555","timestamp":1368153863.008756,"isTest":false,"url":"http://myonlinearcade.com/","ipAddress":"166.13.20.21","userAgent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101       Firefox/20.0","language":"fr","protocolVersion":"0.3","exchange":"appnexus","provider":"appnexus","winSurcharges":{"surcharge":{"USD/1M":50}},"winSurchageMicros":{"surcharge":{"USD/1M":50}},"location":{"countryCode":"CA",          "regionCode":"QC","cityName":"Laval","postalCode":"0","dma":0,"timezoneOffsetMinutes":-1},"segments":{"appnexus":["memberId1357"],"browser":["Mozilla Firefox"],"device_type":["Computer"],"os":["Microsoft Windows 7"]},"userIds":    {"an":"5273283952213481305","xchg":"5273283952213481305"},"imp":[{"id":"156331815539876686","banner":{"w":728,"h":90},"formats":["728x90"]}],"spots":[{"id":"156331815539876686","banner":{"w":728,"h":90},"formats":["728x90"]}]};

and the following commands issued on our Redis instance:

 set  "RTBkit:aug:winSurcharges.surcharge.USD/1M:50"  123.45
 set  "RTBkit:aug:id:85885bb0-b91b-11e2-c4cf-7fba90171555" 9876
 set  "RTBkit:aug:url:http://myonlinearcade.com/" "JSCRIPT"

and two agents configured as following:

   TestAgent agent1(proxies, "bob-the-agent");
   agent1.init();
   agent1.start();
   agent1.configure();
   {
       AugmentationConfig aug_conf;
       aug_conf.name = "redis";
       aug_conf.required = true;
       auto& v =  aug_conf.config;
       Json::Value av(Json::arrayValue);
       av.append("winSurcharges.surcharge.USD/1M");
       av.append("id");
       av.append("exchange");
       av.append("foo.bar"); // not found
       v["aug-list"] = av;
       v["aug-prefix"] = "RTBkit:aug";
       agent1.config.addAugmentation(aug_conf);
   }
   agent1.doConfig (agent1.config);

    TestAgent agent2(proxies, "alice-the-agent");
    agent2.config.account = {"aliceCampaign", "aliceStrategy"};
    agent2.init();
    agent2.start();
    agent2.doConfig(agent2.config);
    {
        AugmentationConfig aug_conf;
        aug_conf.name = "redis";
        aug_conf.required = true;
        auto& v =  aug_conf.config;
        Json::Value av(Json::arrayValue);
        av.append("id");
        av.append("url"); // not found
        v["aug-list"] = av;
        v["aug-prefix"] = "RTBkit:aug";
        agent2.config.addAugmentation(aug_conf);
    }
    agent2.doConfig (agent2.config);

then our RedisAugmentor should produce the following AugmentationList:

[{"account":["aliceCampaign","aliceStrategy"],"augmentation":{"data":{"RTBkit:aug:id:85885bb0-b91b-11e2-c4cf-7fba90171555":"9876","RTBkit:aug:url:http://myonlinearcade.com/":"JSCRIPT"}}},{"account":["testCampaign","testStrategy"], "augmentation":{"data":{"RTBkit:aug:id:85885bb0-b91b-11e2-c4cf-7fba90171555":"9876","RTBkit:aug:winSurcharges.surcharge.USD/1M:50":"123"}}}]