Data logger

In this document we'll write a simple data logger which will be used to log various events emitted by the RTBKit stack to rotating file. While we'll be using files in this example, the interface we'll be using to build our logger is pretty flexible and can be used to store the events into your datastore of choice.

The complete code for the example can be found in the rtbkit repository under rtbkit/examples/data_logger_ex.cc.

Setup

Let's start by creating our data logger object which will listen for log events.

    DataLogger logger(serviceProxies);
    logger.init();
    setupOutputs(logger, logDir, rotationInterval);

The DataLogger class is a simple wrapper to the Logger class which handles the meat of the logging event pub-sub sockets.

Note that the brunt of the setup work is handled in the setupOutputs function which we'll cover in the next section.

    logger.connectAllServiceProviders("adServer", "logger");
    logger.connectAllServiceProviders("rtbRequestRouter", "logger");
    logger.connectAllServiceProviders("rtbPostAuctionService", "logger");

Here we make use of the zookeeper to discover and connect to the various services we want to receive logging events from. In order, these are the:

• Ad server connector which is in charge of receiving win/loss events as well as post-auction events (impressions, clicks, etc.).
• The router which processes bid requests and their associated bid responses.
• The post-auction loop which processes ad server events forwarded from the ad server connector.

Each events we receive through one of these endpoints will be filtered and funnelled into the outputs that we'll setup in the setupOutputs function.

    logger.start();
    while (true) this_thread::sleep_for(chrono::seconds(10));

Once everything is setup, we can start the logger a take a well deserved nap.

Outputs

Here we'll look into the core of the data logger which we'll house in the setupOutputs function.

void
setupOutputs(
        DataLoggerBase& logger,
        const string& logDir,
        const string& rotationInterval)
{

In this function we'll setup two LogOutput object which will be used to write our logging events to files. Note that a log output is analoguous to a sink in other logging frameworks. The two parameters will determine the folder in which we'll dump the log files and the frequency at which we'll rotate the log files.

    auto errorOutput = make_shared<RotatingFileOutput>();
    errorOutput->open(logDir + "/%F/errors-%F-%T.log", rotationInterval);
    logger.addOutput(errorOutput, boost::regex("ROUTERERROR|PAERROR"));

So first up, we'll create a RotatingFileOutput which will create a series of rotating log files that will contain every event that it receives. In the second line we use a format string to specify the location and the name of our log files based on the time at which the file was created. The syntax of the format string should adhere to C++'s strftime function.

The final line is used to register our output with the data logger and specifies the log events we're interested in through a regex filter. In this case we want all error messages coming from the router and the post-auction loop.

    auto strategyOutput = make_shared<MultiOutput>();

The logger framework provides a number of output types which offer enough flexibility to suit whatever scenario you're facing. CallbackOutput, RotatingFileOutput and RotatingCloudOutput are the most common output worth using but if you need something fancier you can always derive the LogOutput class and create your own.

For our next output, we'll get a little fancy with a MultiOutput. This output can be used to regroup multiple outputs under a single output.

    auto createSubOutput = [=] (const string & pattern)
        {
            auto result = make_shared<RotatingFileOutput>();
            result->open(pattern, rotationInterval);
            return result;
        };
	strategyOutput->logTo(
            "MATCHEDWIN",
            logDir + "/%F/$(17)/$(5)/$(0)-%T.log.gz",
            createSubOutput);
    strategyOutput->logTo(
            "",
            logDir + "/%F/$(10)/$(11)/$(0)-%T.log.gz",
            createSubOutput);

Here we house two RotatingFileOutput which are created by the createSubOutput lambda. The interesting bit here is that we use a string generated by by MultiOutput as the file name pattern for our output. This string will be generated by replacing the $(index) placeholders with the value of the log event at that index in the pattern string specified in the second argument to the logTo function. In this case, each event will be placed in a date/campaign/strategy folder structure. Pretty neat.

Finally, whenever MultiOutput receives a log event, it will sequentially check its outputs for a match on a channel which is specified in the first argument of the logTo function. In our example, all MATCHEDWIN events will be redirected to the first output and everything else will be redirected to the second output.

    logger.addOutput(
            strategyOutput,
            boost::regex("MATCHEDWIN|MATCHEDIMPRESSION|MATCHEDCLICK"));
}

Final step for our wonderfully fancy strategyOutput is to add it to our data logger with a filter on auction events generated by the post auction loop.

And that's it. Our logger is complete plus or minus a main function which reads command line arguments. While this example is rather simple, the logger framework is pretty flexible and it's pretty easy to create new outputs to fit whatever logging scheme we can come up with.