Redis Augmentor

The following page introduces RTBkit' Redis Augmentor, its architecture and how to use it.

Introduction

As described in this page, an RTBKIT::Augmentor:

• Receives a series of bid requests and will add information to them in the form of tags or data. This information can be either per account or global to all accounts.
• Subscribes to the Agent Configuration Service to know the augmentation for each client(bidding agent)

The complete source code for the example is available in the rtbkit repository under the following 3 files:

• rtbkit/plugins/augmentor/redis_augmentor.h
• rtbkit/plugins/augmentor/redis_augmentor.cc
• rtbkit/plugins/testing/redis_augmentor_test.cc

This page describes how to write an Augmentor as an instance of the SyncAugmentor. We shall also assume that [accounts](https://github.com/rtbkit/rtbkit/blob/master/common/account_key. h) follow the <campaign>.<strategy> pattern. For more information on accounts, see the banker's documentation.

Architecture

RTBkit's Redis Augmentor is a AsyncAugmentor, which overrides its

void
RedisAugmentor::
onRequest(const AugmentationRequest & request, SendResponseCB sendResponse) /* override */;

is composed of a:

RTBKIT::AgentConfigurationListener agent_config_;

and aggregates or is composed of a

std::shared_ptr<Redis::AsyncConnection> redis_ ;

Given our RTBKIT::AugmentationList, a SendResponseCB is defined as following:

typedef std::function<void (const AugmentationList &)> SendResponseCB;

Redis::AsyncConnection is a thin layer around hiredis, which as far as we are concerned, provides an asynchronous, object oriented API around a physical connection to a Redis instance, with convenient timeout managements, as well as commands pipelining (which are both used in our case). Redis::AsyncConnection has its own events loop, on which it services augmentation callbacks.

It is also important to realise that the inherited AsyncAugmentor, is composed of (amongst other things) a thread pool, which it uses in order to service inbound augmentation requests. The size of this servicing thread poll is a parameter of our base Redis::AsyncConnection; and also as that the life cycle of a bid request consists in sequence of the following events:

1. The exchange sends a bid request to the router where it is parsed by the exchange connector
2. The router relays the bid request to the Augmenter
3. The augmenter returns augmentations("+") to the router
4. The router sends augmented bid request("BR+") to the bidding agent

In the case of our RedisAugmentor, the sequence between 2 and 3 above, can be further specified:

1. The router relays the bid request to the RedisAugmentor
2. For each agent associated with this bid request, the RedisAugmentor tries to fetch an <augmentations>.<redis>.<config>.<aug-list> JSON::ArrayValue from each associated AgentConfigEntry, eventuelly upserting an ordered map jobs indexed by Redis keys, mapping to a set of AccountKey (<campaign>.<strategy>).
3. Out of the keys indexing jobs, the RedisAugmentor then builds a batch of Redis commands, and submits it to its Redis::AsyncConnection member, along with a redis callback defined as a C++11 lambda capturing our job map by value, as well as the SendResponseCB passed as a parameter to the call. Note that these pipelined redis commands are submited with a 4 milliseconds timeout.
4. Upon being called back on the redis thread, our lambda submitted above then constructs an AugmentationList according to both job and the Redis::Results passed as a callback argument by our Redis::AsyncConnection and calls SendResponseCB to pass it back to the calling AugmentorLoop run by our router.

Voilà.

Syntax of the keys fetched from the AugmentationRequest

The general structure of keys which our RedisAugmentor loooks for, is

 prefix:path:value

where prefix is the arbitrary string "RTBkit:aug", and value the value of the attribute of a bid request Json object, as described by a path, using the following syntax:

 - "." => root node
 - ".[n]" => elements at index 'n' of root node (an array value)
 - ".name" => member named 'name' of root node (an object value)
 - ".name1.name2.name3"
 - ".[0][1][2].name1[3]"
 - ".%" => member name is provided as parameter
 - ".[%]" => index is provied as parameter

In our case, the root node needs not to be provided in the key, as it obviously is the bid request in question. For a given agent, the keys are fetched from its associated AugmentationConfig configuration object, as path into the bid request passed along with the AugmentationRequest.

Example

(from rtbkit/plugins/testing/redis_augmentor_test.cc)

The following bid request:

 {"id":"85885bb0-b91b-11e2-c4cf-7fba90171555","timestamp":1368153863.008756,"isTest":false,"url":"http://myonlinearcade.com/","ipAddress":"166.13.20.21","userAgent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101       Firefox/20.0","language":"fr","protocolVersion":"0.3","exchange":"appnexus","provider":"appnexus","winSurcharges":{"surcharge":{"USD/1M":50}},"winSurchageMicros":{"surcharge":{"USD/1M":50}},"location":{"countryCode":"CA",          "regionCode":"QC","cityName":"Laval","postalCode":"0","dma":0,"timezoneOffsetMinutes":-1},"segments":{"appnexus":["memberId1357"],"browser":["Mozilla Firefox"],"device_type":["Computer"],"os":["Microsoft Windows 7"]},"userIds":    {"an":"5273283952213481305","xchg":"5273283952213481305"},"imp":[{"id":"156331815539876686","banner":{"w":728,"h":90},"formats":["728x90"]}],"spots":[{"id":"156331815539876686","banner":{"w":728,"h":90},"formats":["728x90"]}]};

the following commands issued on our Redis instance:

 set  "RTBkit:aug:winSurcharges.surcharge.USD/1M:50"  123.45
 set  "RTBkit:aug:id:85885bb0-b91b-11e2-c4cf-7fba90171555" 9876
 set  "RTBkit:aug:url:http://myonlinearcade.com/" "JSCRIPT"

and two agents configured as following:

    TestAgent agent1(proxies, "bob-the-agent");
    agent1.init();
    agent1.start();
    agent1.configure();
    {
        AugmentationConfig aug_conf;
        aug_conf.name = "redis";
        aug_conf.required = true;
        auto& v =  aug_conf.config;
        Json::Value av(Json::arrayValue);
        av.append("winSurcharges.surcharge.USD/1M");
        av.append("id");
        av.append("exchange");
        av.append("foo.bar"); // not found
        v["aug-list"] = av;
        v["aug-prefix"] = "RTBkit:aug";
        agent1.config.addAugmentation(aug_conf);
    }
    agent1.doConfig (agent1.config);

and:

    TestAgent agent2(proxies, "alice-the-agent");
    agent2.config.account = {"aliceCampaign", "aliceStrategy"};
    agent2.init();
    agent2.start();
    agent2.doConfig(agent2.config);
    {
        AugmentationConfig aug_conf;
        aug_conf.name = "redis";
        aug_conf.required = true;
        auto& v =  aug_conf.config;
        Json::Value av(Json::arrayValue);
        av.append("id");
        av.append("url"); // not found
        v["aug-list"] = av;
        v["aug-prefix"] = "RTBkit:aug";
        agent2.config.addAugmentation(aug_conf);
    }
    agent2.doConfig (agent2.config);

then our RedisAugmentor should produce the following AugmentationList:

[{"account":["aliceCampaign","aliceStrategy"],"augmentation":{"data":{"RTBkit:aug:id:85885bb0-b91b-11e2-c4cf-7fba90171555":"9876","RTBkit:aug:url:http://myonlinearcade.com/":"JSCRIPT"}}},{"account":["testCampaign","testStrategy"], "augmentation":{"data":{"RTBkit:aug:id:85885bb0-b91b-11e2-c4cf-7fba90171555":"9876","RTBkit:aug:winSurcharges.surcharge.USD/1M:50":"123"}}}]