Overview of developing an Exchange Connector

Architecture

This page on the RTBKit wiki describes the architecture of RTBKit. It also provides a summary description of the function of each major component of the platform.

Development Assumptions

This documentation assumes that you are an exchange or SSP who wants to create an RTBkit Exchange Connector to allow your inventory to be made available to RTBkit deployments.

For this tutorial, we make the following additional assumptions about how your system will integrate with RTBKit:

• Your exchange communicates via HTTP and will connect to a URL of an HTTP server on the internet to make inventory available
• Your exchange sends bid requests and receives bid responses in either JSON or Google protocol buffers format
• Your exchange can create a limited number of connections per seat so as to avoid overloading a single system

Finally, we assume you have a working knowledge of Linux and C++ development, because these are the technologies used to create and deploy an RTBKit Exchange Connector.

Exchange Connectors must be implemented in C++, because exchange connectors live in-process with the Router. They must derive from the ExchangeConnector base class. In addition, we recommend native processes rather than processes running in managed runtimes with garbage collection because this can interfere with the Router’s ability to respond in a timely manner to all bid requests.

Creating a Development Environment

To create a local development environment and a working test instances of RTBKit, follow the instructions here.

Using Google protocol buffers to communicate with an exchange

The adx exchange connector was the first exchange connector built with support for Google protocol buffers.

The adx protobuf definition file rtbkit/rtbkit/plugins/exchange/realtime-bidding.proto was obtained from https://developers.google.com/ad-exchange/rtb/downloads/realtime-bidding-proto.txt. The protobuf definition file defines some data structures with fairly generic names like BidRequest and BidResponse. Unfortunately this protobuf definition file does not use a package declaration as suggested in https://developers.google.com/protocol-buffers/docs/cpptutorial. This means that if you add another protobuf definition file for a different exchange, and if it also happens to use the same generic names like BidRequest and BidResponse without a package declaration, you will have a name conflict.

Since the adx protobuf definition file was the first one added, it should be left alone and a package declaration statement should be added to the protobuf definition file for any new exchange that has a name conflict with adx. It will then be necessary to merge this package declaration statement into any updated version of the protobuf definition file.

Generating Bid Requests to Work With

This section of the RTBKit wiki explains how to set up a complete working instance pointing at a mock exchange and recieving synthetic but realistic bid requests.

Development Overview

You implement your Exchange Connector by creating a derived class from HttpExchangeConnector that contains your bid request handling logic. In general, think of this class as encapsulating all the integration points between your bidder and a particular exchange which can't be generalized, because they vary by exchange or allow you to customize the business logic of your Connector.

Perhaps the best way to understand how to implement your Connector is to review the code, following the class hierarchy from the ExchangeConnector base class to the HttpExchangeConnector base class and then to one of the production-ready derived Connectors that ships with RTBKit, such as RubiconExchangeConnector. Review the code and doc comments to understand the methods' purpose.

You define your custom Connector logic by implementing virtual methods. Some of these methods are given more full treatment elsewhere in this documentation. In particular review "How to configure an Exchange Connector", "How to write an Exchange Connector bid request parser" and "How to implement an Exchange Connector bid response generator".

Other methods are discussed in the following sections.

Override: Time Available

The Router needs to know how long it has to process your bid request. This information is normally provided in one of three ways:

1. It is a constant value that is known ahead of time from the exchange
2. It is included in the bid request
3. It is included in the HTTP headers for the bid request

You will need to override the following method in order to return the value. Note that this method should not decode the entire bid request; it should instead attempt to extract the time available with the smallest amount of work possible. This is because the router may return no-bid responses immediately to requests that have very little chance of being processed in time, in order to shed load when it is overloaded.

Here is the method in the HttpExchangeConnector base class.

/** 	Return the available time for the bid request in milliseconds.
		This method should not parse the bid request, as when 
		shedding load we want to do as little work as possible.

		Most exchanges include this information in the HTTP headers.
*/
virtual double
getTimeAvailableMs(const HttpHeader & header, const std::string & payload);

RTBKit ships with a derived ExchangeConnector for the Rubicon exchange. Here is an example of a concrete implementation of the method. Note that the implementation strives to be as efficient as possible.

double
RubiconExchangeConnector::
getTimeAvailableMs(HttpAuctionHandler & connection,
               	   const HttpHeader & header,
               	   const std::string & payload)
{
	// Scan the payload quickly for the tmax parameter.
	static const string toFind = "\"tmax\":";
	string::size_type pos = payload.find(toFind);
	if (pos == string::npos)
   		return 10.0;
    
	int tmax = atoi(payload.c_str() + pos + toFind.length());
    
	return tmax;
}

Override: Dropped Auction Response

This method should return a no-bid to the exchange. Here is the signature of the method in the HttpExchangeConnector base class.

/** 	Return a stringified JSON of the response for when we 
		drop an auction.
*/
virtual HttpResponse 
getDroppedAuctionResponse(const Auction & auction, const std::string & reason) const;

Here is an implmentation in the RubiconExchangeConnector that ships with RTBKit.

HttpResponse
RubiconExchangeConnector::
getDroppedAuctionResponse(const HttpAuctionHandler & connection,
                      	 const Auction & auction,
                      	 const std::string & reason) const
{
	return HttpResponse(204, "application/json", "{}");
}

Override: Bid Request Error Response

Again, the base class signature …

/** 	Return a stringified JSON of the response for our auction.
		Default implementation calls getResponse() and stringifies 
		the result.

 		This version is provided as it may be more efficient in 
		terms of memory allocations.

		The first element returned is the HTTP body, the second is the
		content type.
*/
virtual HttpResponse
getErrorResponse(const Auction & auction, const std::string & errorMessage) const;

… and the concrete implementation from the Rubicon Connector.

HttpResponse
RubiconExchangeConnector::
getErrorResponse(const HttpAuctionHandler & connection,
             	 const Auction & auction,
             	 const std::string & errorMessage) const
{
	Json::Value response;
	response["error"] = errorMessage;
	return HttpResponse(400, response);
}

Optionally Override: Campaign Compatibility

These methods have a default implementation inExchangeConnector but you can override them in your derived class to customize behavior. The comment in the code shown here documents the functionality. These method let you handle logic specific to the interaction of the campaigns you intend to bid on and the exchange this Connector is integrating with. Given that, the default implementation is to simply allow all campaigns to bid. Note that any other implementation will potentially make your Connector less generally useful to bidding organizations other than yours.

/** Given an agent configuration, return a structure that describes
    the compatibility of each campaign and creative with the
    exchange.

    If includeReasons is true, then the reasons structure should be
    filled in with a list of reasons for which the exchange rejected
    the creative or campaign.  If includeReasons is false, the reasons
    should be all empty to save memory allocations.  Note that it
    doesn't make much sense to have the reasons non-empty for creatives
    or campaigns that are approved.

    The default implementation assumes that all campaigns and
    creatives are compatible with the exchange.
*/
virtual ExchangeCompatibility
getCampaignCompatibility(const AgentConfig & config, bool includeReasons) const;

/** Tell if a given creative is compatible with the given exchange.
    See getCampaignCompatibility().
*/
virtual ExchangeCompatibility
getCreativeCompatibility(const Creative & creative, bool includeReasons) const;

The RubiconExchangeConnector shows one flexible and practical approach to implementing another version of the exchange compatibility method -- reading in settings from a config file.

ExchangeConnector::ExchangeCompatibility
RubiconExchangeConnector::
getCampaignCompatibility(const AgentConfig & config,
                         bool includeReasons) const
{
    ExchangeCompatibility result;
    result.setCompatible();

    auto cpinfo = std::make_shared<CampaignInfo>();

    const Json::Value & pconf = config.providerConfig["rubicon"];

    try {
        cpinfo->seat = Id(pconf["seat"].asString());
        if (!cpinfo->seat)
            result.setIncompatible("providerConfig.rubicon.seat is null",
                                   includeReasons);
    } catch (const std::exception & exc) {
        result.setIncompatible
            (string("providerConfig.rubicon.seat parsing error: ")
            + exc.what(), includeReasons);
       return result;
    }

   result.info = cpinfo;

   return result;
}

Optionally Override: Creative Compatibility

You can similarly configure the bidder to not respond with certain creatives in certain situations, based on your custom logic. You do this by implementing getCreativeCompatibility /** Tell if a given creative is compatible with the given exchange. See getCampaignCompatibility(). */ virtual ExchangeCompatibility getCreativeCompatibility(const Creative & creative, bool includeReasons) const;

Again, the RubiconExchangeConnector is a good place to review in existing implementation of this method.

Override: Accept Bid Request Probability

Similarly, this method controls connector business logic that may be particular to your use case, because it sets the percentage of bid requests from the exchange that the Connector will pass through to your RTBKit bidder. You must implement this in your derived connector.

/** Set which percentage of bid requests will be accepted by the
    exchange connector.
*/
virtual void setAcceptBidRequestProbability(double prob) = 0;

Win Price Decode

Exchanges transmit winning prices in different units, currencies, encodings and encryption schemes. Your Connector may need to let users decode win prices. This is not part of the base interface at this time but the RubiconExchangeConnector adds a method to handle this, providing an example.

Note that you should be very careful dealing with precision for methods and variables related to currency units, because different exchanges require and use different levels of precision. For example, this method uses float but AppNexus transmits prices as double values, which would truncate if assigned to float variables.

float
RubiconExchangeConnector::
decodeWinPrice(const std::string & sharedSecret,
           	const std::string & winPriceStr)
{
	ExcAssertEqual(winPriceStr.length(), 16);
    
	auto tox = [] (char c)
    	{
        	if (c >= '0' && c <= '9')
            	return c - '0';
        	else if (c >= 'A' && c <= 'F')
            	return 10 + c - 'A';
        	else if (c >= 'a' && c <= 'f')
            	return 10 + c - 'a';
        	throw ML::Exception("invalid hex digit");
    	};

	unsigned char input[8];
	for (unsigned i = 0;  i < 8;  ++i)
   		input[i] = tox(winPriceStr[i * 2]) * 16 + tox(winPriceStr[i * 2 + 1]);
    
	CryptoPP::ECB_Mode<CryptoPP::Blowfish>::Decryption d;
	d.SetKey((byte *)sharedSecret.c_str(), sharedSecret.size());
	CryptoPP::StreamTransformationFilter
   		filt(d, nullptr,
       	  CryptoPP::StreamTransformationFilter::NO_PADDING);
	filt.Put(input, 8);
	filt.MessageEnd();
	char recovered[9];
	size_t nrecovered = filt.Get((byte *)recovered, 8);

	ExcAssertEqual(nrecovered, 8);
	recovered[nrecovered] = 0;

	float res = boost::lexical_cast<float>(recovered);

	return res;
}

Optionally Override: Additional Methods

There are a few other minor methods you need to understand and implement in your Connector. Again, the basic pattern is to follow the class hierarchy from the ExchangeConnector base class to the HttpExchangeConnector base class and then to one of the derived concrete Connectors that ships with RTBKit, such as RubiconExchangeConnector.

/** Return the name of the exchange, as it would be written as an
    identifier.
*/
virtual std::string exchangeName() const = 0;

/** Set the time until which the exchange is enabled.  Normally this will
    be pushed forward a few seconds periodically so that everything will
    shut down if there is nothing controlling the exchange connector.
*/
virtual void enableUntil(Date date) = 0;