Agent Configuration and Filtering

The Agent Configures the Core

An RTBkit Agent, upon bootup, sends a 'config' message into the RTBkit Core, specifically to the Agent Configuration Service, whose function is to rebroadcast the agent configuration message to any other component that needs it. The key RTBkit Core components which consume agent configuration messages are the Router, Post-Auction Loop and Augmenters:

• The Router, upon receiving an agent configuration message, will start sending bid requests to the agent, according to the filtering rules in the configuration.
• The Post-Auction Loop will relay post-auction messages (wins, impressions, clicks, conversions) to the agent.
• Augmenters add per-agent augmentations to bid requests as per the agent configuration message.

In this way, the Agent is configuring the Core so as to be able to receive a tailored stream of events. By analogy, the Agent is like a customer and the Core is like a restaurant: the Agent tells the restaurant exactly what it would like to be served.

Sample Agent Configuration Message

For reference in the rest of this section, here is a sample JSON-ified agent configuration message:

{
	"account": ["parent", "child"],
	"bidProbability": 0.1,
	"creatives": [ 
		{"id": 1, "width": 300, "height": 250}, 
		{"id": 2, "width": 160, "height": 600} 
	],
	"languageFilter": { "include" : ["en"], "exclude": [] },
	"segmentFilter": {
		"sample1": { "include" : [], "exclude": ["bad"] },
		"sample2": { "include" : ["good"], "exclude": [] },
		"colors": { "include" : ["blue", "red"], "exclude": ["black", "white"] }
	},
	"augmentations": {
		"freq-rec": {
			"required": true,
			"config": {"maxPerDay": 10},
			"filters": { "include": [], "exclude": ["too-many"] }
		}
	},
	"maxInFlight": 10
}

Filters

The RTBkit core receives, via its Exchange Connector, a stream of bid requests. Each Agent, via its configuration message, informs the Core about which subset of the bid request stream it would like the Core to relay. Every time the Router receives a bid request from an Exchange Connector, it goes through its matching logic to figure out which Agents it should relay the request to. The matching looks at the following chunks of the agent configuration: bid probability, creatives, static filters, augmenter filters and dynamic filters.

As a general note, a bid request must pass all of the filters to be relayed to the Agent. For the filters defined as include/exclude pairs, the logic is: the bid request passes the filter if it matches any of the values in the include list and none of the values in the exclude list.

For example, for the configuration above, there are 4 filters (not including creative matching, which is covered below): languageFilter, segmentFilter.sample1, segmentFilter.sample1, segmentFilter.colors, and augmentations.freq-rec.filter, and a bid request must pass all of these filters to be relayed to the Agent. The filters are all include/exclude pairs, so they follow the any/none logic above. The bid request therefore passes if this pseudo-code returns true:

all(
  any(
    'en' in bidRequest.language
  ), 
  none(
    'bad' in bidRequest.segments.sample1
  ), 
  any(
    'good' in bidRequest.segments.sample2
  ), 
  any(
    'blue' in bidRequest.segments.colors, 
    'red' in bidRequest.segments.colors
  ), 
  none(
    'black' in bidRequest.segments.colors, 
    'white' in bidRequest.segments.colors
  ), 
  none(
    'too-many' in bidRequest.augmentations.freq-rec.tags 
  )
)

Creatives

An Agent declares to the Core which creatives it is willing to bid with, and the Core will only relay bid requests for which the Agent has at least one compatible creative (where compatibility refers at the very least to dimensions, but possibly other factors). The Exchange Connector can also enforce business rules that matter to the the supply source, for example some exchanges deem certain types of creatives to be objectionable for certain bid requests.

In the example above, the Agent will only receive bid requests which are for IAB 'medium rectangle' or 'wide skyscraper' units.

HTML/Markup/JS etc

Within the Core, the Exchange Connector is responsible for parsing incoming bid requests from a specific exchange, in that exchange's format, into the RTBkit internal bid request object. It is also responsible for the inverse operation: assembling bid responses in the exchange-specific correct format out of bids from Agents.

When an Agent's bid (which refers to a specific creative in this Agent's list of creatives) wins the internal auction within the Core, the Exchange Connector will get all the information related to this bid, including the creative configuration specific to the creative referred to by the bid. If the supply source that the Exchange Connector connects to requires, for example, some HTML markup for the creative, then the creative configuration will need to contain this markup. The proper place for this is in a block called providerConfig, which is a dictionary whose keys should be the names of the Exchange Connectors (see below).

Supply-source-specific creative matching

The Exchange Connector is also responsible for matching creatives to supply source filtering rules (i.e. publisher constraints). For example, if a creative contains references to alcohol, and a publisher doesn't wish to have such creatives on their site, the bid request may contain some filtering directives, and the Exchange Connector is responsible for implementing this filtering. Creative-level attributes that feed into this filtering logic can also be passed to the Exchange Connector via the providerConfig block.

Creative configuration sample with providerConfig block

	...
	"creatives": [ 
		{
			"id": 1, "width": 300, "height": 250,
			"providerConfig": {
				"supplySourceX": {
					"markup": "markup goes here",
					"attributes": ["alcohol"],
					...
				},
				...
			} 
		}, 
		...
	],
	...

Static Filters

There are two kinds of static filters: RTBkit predefined filters, such as location, language etc, and 'segment' filters, which are Exchange Connector-dependent.

Predefined Filters

RTBkit defines hardcoded filters for the following fields in the bid request object, provided that the bid request parser has populated those fields:

• hosts and urls
• languages and locations
• exchanges
• position relative to the page's fold
• hour of the week

In the example above, the Agent will only receive bid requests if the language field contains en.

####Segment Filters

RTBkit allows for arbitrary filtering based on 'segments'. Segments are values which are set by the bid-request parser at parse-time. For example, a parser could look at a bid request and decide that for a segment called x, the bid request has values y and z.

In the example above, the Agent will receive bid requests which don't have a value bad for a segment called sample1 and do have a value good for a segment called sample2. Furthermore, the segment called colours needs to have either blue or red to pass, but even if it does, the request will not reach the agent if black or white is in the colors segment.

Bid Probability

A key part of pacing and load management is to not bid on every single possible bid request, so an Agent can ask of the Core to only pass it a certain fraction of the elligible bid requests. This is the bid probability.

The bid probabiliy in the example above is 0.1, so the Agent will only receive 10% of the bid requests which matched the previous filters.

Augmenter Filters

RTBkit Augmenters are components which the Core calls upon to add data to bid requests. Data is added on a per-Agent basis. An Agent requests that the Core pass its bid requests through a given Augmenter via its configuration message. Beyond adding arbitrary data to the bid request, an Augmenter can add 'tags' that an Agent can instruct the Core to use for filtering.

In the example above, the Agent will only receive bid requests which have been augmented by an Augmenter called freq-rec and furthermore, which that Augmenter has not tagged with too-many. Note that the configuration for the Augmenter specifies max=10, which in the context of this fictional Augmenter, could mean that the Augmenter should tag requests with too-many if this Agent has already shown 10 impressions.

Dynamic Filters

Finally, the Core can apply some dynamic filters to a bid request. For example, an Agent can tell the core that it can only operate on a certain number of bid requests simultaneously, and the Core will honour this by only keeping this number of bid requests "in flight" at any one time.

In the example above, the Agent will only receive a max of 10 bid-requests at any one time, and will not receive any more until it responds to one of the 10.

How do I...?

Given the basic rules above, there are a number of questions which come up repeatedly:

How do I apply more than one rule at a time?

For example, if I wanted to apply above-the-fold filtering only at certain times of the day?

At the moment, this is not supported: filters are applied independently and ANDed together. That said, an augmenter could be used to implement specific cases.

How do I filter on X (i.e. only match 'males in Germany')?

• If X is present in the bid request received by the Exchange Connector, then the bid request parser should set some field in the bid request object.
  • If X is a 'standard'/'predefined' field such as language or location, this is very straightforward and a predefined filter can be used.
  • If X is something implementation-specific for which there is no predefined field, it can be stored as a 'segment' and a segment filter can be used.
• If X is not present in the original bid request but is available in a data store, and the key for the data store is available in the bid request, then an augmenter is required to look up X based on some piece of the bid request, and an augmenter-filter can then be used.

For example, if X is 'males in Germany', here is how these rules would apply:

• Geo-location data is usually present in the bid request, and should be parsed by the Exchange Connector into the standard location fields in the bid request object, so the predefined locationFilter mechanism can be used.
• Gender data is usually not present in the bid request, and so an augmenter would have to be built, which sets "genderMale" or "genderFemale" tags wich could then be used by augmenter-filters.
• If gender data is present in the bid request, the Exchange Connector could parse that data into a segment and the segmentFilter mechanism could be used.

How do I user regular expressions for location filtering?

When a bid request is received there's two level of location filtering that is applied by the router:

• per agent config
• per creatives within the agent config.

The filter is specified as an include/exclude filter which accepts regexes to be match against the following string: COUNTRY:REGION:CITY:POSTAL:DMA:METRO .

Here's an example:

{
    ...
    "locationFilter": { "include": [ "^CA", "^US" ], "exclude": [ "^US:NY" ] },
    "creatives": [
        { ... "locationFilter": { "include": [ "^US" ] },
        { ... "locationFilter": { "include": [ "^CA" ] }
    ],
    ...
}

The top level filter in this example will only allow bid requests that originates from Canada the US but not from New-York. If this top level filter passes, each creatives will then be evaluated. The first creative will only be available for bidding if the bid request originates from the US and so on for the second creative.

Note that while you could use far more complex regular expressions, this would significantly impact the performance of the filters and could drastically reduce the throughput of the router. It's better to keep the regexes as simple as possible.

If you want to be able to create the config in C++-land. The following should work:

    config.locationFilter.include = { L"^CA", L"^US" };
    config.locationFilter.exclude = { L"^US:NY" };