Multi language bidding agent

Jan Sulmont edited this page Apr 30, 2014 · 20 revisions

In this document, we shall present two distinct approaches to a multi-language agent for RTBbit. But since they both rely on a somewhat simplified interface to the bidding agent, we shall touch first about the advantages to rolling out a simplified C++ interface to the bidding agent -- and since these two sub projects are interrelated, perhaps conclude that this multi language agent project is an occasion to kill two birds with a same stone.

simplified C++ interface to the bidding agent.

The RTBkit framework is a complex framework involving many advanced concepts, which the user of the framework often have pain to apprehend. In a way, a majority of our users are mostly concerned about business matter related to realtime bidding in electronic/algorithmic advertising. And just because this topic involves advanced computing notions, does not imply that our users should master them all. Similarly, users of a modern no-sql engine (such as Cassandra) are not expected to know everything about distributed consistency.

This remark also holds for the C++ level required in order to apprehend RTBkit: it should be possible to use the framework without having to be a medium to advanced C++ developer.

need for a multi language bindings to the bidding agent.

If C++ is wildly used in Finance, the advertisement business is somewhat tainted with web-related technologies, such as php or node-js etc. Another incentive to multi-language bindings, is the need to quickly roll out new bidding strategies, would it only be for the sake of testing/experimenting as it is often the case in finance: trading strategies are often prototyped and tested in Python, R or Matlab, and only then converted to C++.

Classically, every possible "exit" language (python, c#, java, ruby, ...) has some kind of C (or even C++) native API, which makes it possible to go back and forth from the exit to C/C++.

However, from the software engineering perspective, this may result in as many specific sub-projects, as there are exit languages considered. This altogether make the case for considering third party softwares facilitating this task, such as SWIG, which we shall only consider as a solution thereafter.

simplified C++ interface: description.

A simplified C++ interface, should capture whatever is needed for an agent to interact with a router, and nothing more, letting obviously aside matters related to implementation choices/necessities, dependencies, and making as little assumptions as possible on the target computing environment. It is also in our interest to keep the data types involved as close as possible to C/C++ primitive data types and plain old data structures.

Basically, a bidding agent performs the following operations.

  • "discovers" the various bits belonging to its performing environment.
  • identifies itself against a router, and specifies its subscription needs (interested in these bid requests only)
  • receive notifications from the router: bid requests, bid results, deliveries notification, error notifications.
  • occasionally place bids against the router.

From this respect, it is an I/O automaton, exchanging (sending and receiving) messages with its environment:

  • receives a "proxy" message describing its running environment
  • sends a "config" message describing its performing context (configuration) to the router
  • receives notification messages: "bid-request", "bid-resut", "delivery", "error" from the router
  • sends "bid-response" messages to the router

Each of these messages is captured either by a string or a collection of strings representing JSON objects, which along with the verbs of the interface, are described in this piece of code.

simplified C++ interface: asynchronous.

The current C++ bidding agent is half asynchronous: it sends synchronously to the router, and receives asynchronously from the router (more precisely, it sends synchronously its bid-response to a local queue, which are the processed asynchronously by the MessageLoop). Likewise, the first proposed simplified C++ interface is asynchronous: messages from the router are received via dedicated callbacks, also as described here.

swig file.

Basically, SWIG a kind of pre-compiler, which takes a swig-file as an input, and generates whatever stubs are needed to build whatever software layer, in order to connect to the C/C++ code described by the swig-file, from the chosen exit language. Note that it is not a one way journey, as calls can originate from either the C/C++ or the exit language. Ideally, the swig file needs to be written only once for all the exit languages considered. Here is the swig file used for our asynchronous lightweight RTBkit interface.

Cross language polymorphism.

Recent versions of SWIG provide for a feature called cross language polymorphism. This allows, in the exit language targeted, to override virtual member functions defined in C++. This of course, assumes that the exit language offers enough support for OO (unlike LUA, which does not support this feature).

In our case, the simplified C++ interface defines one callback object per message type, each with a virtual method call. This call back object needs to be extended in the exit language and the extension needs to override the inherited call method. The simplified C++ interface also exposes a doBid method, which can be called from the exit language (typically in the the call method of the extended BidRequest).

Example Python

The tree contains the various bits needed in order to run the a python bidder written against the swig file discussed above. A quick python example for a python bidder working with our current demo stack can be found here. Once the build is done, all you have to do in order to run the example, is:

   export PYTHONPATH=`pwd`/build/x86_64/bin
   python rtbkit/examples/bidder.py

It is also possible to run the bidder from the demo stack, as explained here. For this to work, in that case, in rtbkit/sample.launch.json you need to replace:

        {
           "name": "fixed-price-agent",
           "root": ".",
           "path": "build/x86_64/bin/bidding_agent_ex",
           "arg": [
              "-B", "rtbkit/sample.bootstrap.json"
            ],
           "log": true
         }

by:

        {
           "name": "fixed-price-agent",
           "root": ".",
           "path": "rtbkit/examples/bidder.py",
           "log": true
         }

Please note that, the only things needed in order to use the simplified C++ library, are one header file (lwrtb.h) and one shared object (libwrtb.so). The header only includes standards C++ files. The shared object doesn't require any other library but classic C and C++ runtime. Building an agent, does not require any building system, such as the one found in RTBkit. Further, all you need in order to build another SWIG exit language, once swig installed on your box, is both the header and the library, along with the swig file.

Building a similar example using Java, typically takes the following steps:

swig -java -c++ lwrtb.i
g++ -pthread -std=c++11 -I /usr/local/jdk1.7.0_45/include -I /usr/local/jdk1.7.0_45/include/linux -fPIC -I.. -g -c lwrtb_wrap.cxx
g++ -pthread -std=c++11 -shared -o libLWRTB.so lwrtb_wrap.o ./liblwrtb.so
/usr/local/jdk1.7.0_45/bin/javac *.java
/usr/local/jdk1.7.0_45/bin/java -Djava.library.path=. runme

The content of runme.java is:

// quick and dirty java POC
// just override one callback 
public class runme 
{
    final static String agent_config = "... as in examples/bidder.py ... ";
    final static String proxy_config = "... as in examples/bidder.py ... ";
    static { System.loadLibrary("LWRTB"); }
    public static void main(String argv[]) 
    {
	class MyBRcb extends BidRequestCb
	{
	    public void call(Bidder arg0, double ts, 
                         String id, String br, 
                         String bids, double timeleft, 
                         String augs, String wcm) 
        {
		   System.out.println("MyBRcn::call "+ts+" " + id + " " + bids);
	    }
	}
	System.out.println("hello");
	Bidder bob = new Bidder("bob", proxy_config);
	BidRequestCb br_cb = new MyBRcb();
	DeliveryCb dv_cb = new DeliveryCb();
	ErrorCb er_cb = new ErrorCb();
	BidResultCb res_cb = new BidResultCb();
	bob.setBidRequestCb (br_cb);
	bob.setDeliveryCb (dv_cb);
	bob.setErrorCb (er_cb);
	bob.setBidResultCb (res_cb);
	bob.init();
	bob.doConfig (agent_config);
	bob.start(true);
    }
}
Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.