Intent to Implement: Real Time Config 2

[bradfrizzell]

Real Time Config 2.0

Objective

For AMP Fast Fetch, support publisher-specified, multiple, simultaneous callouts in order to augment targeting information included on the ad request. See original RTC I2I #8551.

Background

Remote HTML support was added for delayed fetch to support first party cookie targeting given AMP does not allow custom javascript and pages served from the AMP cache are identical for each user. However, its implementation leverages the fact that custom javascript execution occurs as a part of generating an ad request. Fast fetch allows for sending the ad request early by moving all code related to ad generation within the AMP runtime and is therefore incompatible with remote HTML. Real Time Config (RTC) was added to DoubleClick to allow for per-user targeting, see the initial I2I #8551 and the initial implementation #9857 . In order to support augmenting queries with targeting data from multiple sources, we propose modifying the design to allow multiple call-outs executed per slot.

Overview

The new design of RTC is per-slot, with a maximum of 5 parallel call-outs allowed per slot. RTC will be usable by any Fast Fetch network implementation. Call-outs for all slots will be sent as soon as possible. There are two different types of callouts that will be supported (see Example section below):

1. Custom URL callout

• The publisher specifies a custom URL that should be called out to. For example, a publisher may specify their own targeting server as a custom URL.

2. Vendor-Specified URL

• RTC will now support call-outs to third-party vendors. For example, take VendorFooBar which provides a service that returns similar interests when provided a given interest (i.e. “baseball” yields [“sports”, “apple-pie”]). If VendorFooBar wants publishers to be able to use them for RTC call outs, they simply add their call-out url with built in macros to the AMP RTC vendor registry. Then publishers specify that they want to call out to VendorFooBar, and supply the value to substitute into the macro. This gives the Vendor complete control over the actual URL, with the Publisher only needing to supply the relevant inputs.

In both cases, the results of these call-outs will be passed to the Fast Fetch implementations as part of ad url construction.

Requirements

• Requests are made in parallel per slot as soon as possible
• Limit to at most 5 call-outs per slot
• The callout urls are deterministic purely by the declaration within the page, and all URLs may utilize macro substitution. No other client-side logic or serial call-outs (cannot use result of one callout to build the request of another).
• Logic to merge the responses from the callouts will be specified by the fast fetch network implementation.
• Publisher can use a Vendor-Specified url such that url construction can be controlled by that vendor. Publisher can also provide a Custom Url to support first party call-out.
• Default timeout of 1 sec, which can be tightened by publisher. Publisher can indicate how to handle timeout/network failure, default being to send ad request with whatever information is available by the timeout, i.e. information from late responses is silently dropped.
• RTC is opt-in for both Publishers and Ad Networks, and those that choose not to use it will not experience file size increase.

Design

AmpA4A Modifications

Only Fast Fetch extensions are eligible for Real Time Config, and can opt-in by importing real-time-config-manager.js and modifying getAdUrl to consume the results.

If a publisher then wishes to utilize RTC on their page, they must add a valid RTC configuration to their amp-ad elements.

Both a valid Fast Fetch network and a valid publisher configuration are checked before RTC is executed on a page:

/** amp-a4a.js */

/**
 * @return {boolean} whether RTC is valid for the given slot. 
 */
boolean supportsRealTimeConfig_() {
    return !!AMP.RealTimeConfigManager && <valid config>;
}

Publishers specify callout targets at the slot level on the amp-ad tag via ‘rtc-config’ parameter. Additionally, publishers can specify a timeout value,of 1 second or less (default value is 1 second). Here is an example, with the JSON object expanded for readability:

/** some_pub_page.html */
...
<amp-ad type=doubleclick rtc-config=”{
    "vendors": {
       “vendorFooBar”: {
          “placementId”: 12345
          “someOtherThing”: “abcdef”
       }
    },
    “urls”: [“https://pub.com/rtc-endpoint”],
    “timeoutMillis”: 500
}”...></amp-ad>
...

Specification for rtc-config attribute:

• Attribute must be valid JSON.
• If neither “vendors” or “urls” is specified, no RTC will be performed for that ad slot.
• Keys:
  • “vendors”
    • Optional parameter.
    • Value must be a map of macro replacement keys to macro replacement values.
    • Macro replacement keys must be strings, and are case sensitive.
  • “urls”
    • Optional parameter.
    • Value is an array of URL strings.
    • All URLs must be valid.
    • Url endpoints must use HTTPS.
    • Urls may also utilize macro replacement, but only if the fast fetch network supports it.
  • “timeoutMillis”
    • Optional parameter.
    • Must be less than or equal to 1000.
    • Timeout is specified in milliseconds.
    • If unspecified, default is to use a timeout of 1000ms.
    • If specified value is greater than 1000ms, a timeout of 1000ms will be used.

In order to spare publishers the details of having to construct urls for external vendors, vendors may register a url with macros in a central file called callout-vendors.js, which maps unique vendor names to urls. The url for a vendor will be expanded, given custom values provided by the publisher, prior to callout. For instance:

/** callout-vendors.js */
vendors: {
   “vendor1”: “https://vendor1.com/foo?make=call&customer=CUSTOMER_ID”,
   “vendor2“: “https://vendor2.com/blah?exec=true&partnerId=PLACEMENT_ID“
};

Publishers may also utilize macro substitution for non-vendor urls. In this case, instead of macro keys being specified by a vendor and values being specified by the publisher, both the keys and values are specified by the ad network overriding getCustomCalloutMacros in their Fast Fetch implementation.

For instance, take this example allowing SIZE to be inflated in call-out urls:
First, the ad network overrides getCustomCalloutMacros:

/** amp-ad-network-doubleclick-impl.js */ 

/**
 * Get custom expansion macros to be applied to call-out urls.
 * @return {!Object<string, !src/service/impl/variable-source#SyncResolverDef>}
 */
getCustomCalloutMacros() {
   return {
     SIZE: () => this.element.getAttribute(‘height’) + ‘x’ + this.element.getAttribute(‘width’),
   };
}

Then a publisher specifies the macro in their RTC url as follows:

/** some_pub_page.html */
...
<amp-ad type=doubleclick rtc-config=”{
    “urls”: [“https://pub.com/rtc-endpoint?size=SIZE”],
}”...></amp-ad>
...

The RealTimeConfigManager will then merge the results of getCustomCalloutMacros into the publisher specified URL, which in this example, with an ad size of 320x50, yields https://pub.com/rtc-endpoint?size=320x50

One XHR CORS request will be made to each endpoint specified, with all requests happening in parallel. Upon completion of callouts, the RealTimeConfig manager will pass getAdUrl an array of all the response objects, formatted as:

  { ‘url’: <callout url>,
    ‘response’: <response value>,
    ‘delayMillis’: <time the callout took>,
    ‘error’: <error message>},
 …
]

• url
  • Callout URL
• response
  • Optional parameter, only included if response succeeded
  • Response from the endpoint called out to
• delayMillis
  • Time, in milliseconds, that the RTC callout took to complete
• error
  • Optional parameter, only included if response was unsuccessful
  • Error message to indicate what went wrong

Note that in the event of a timeout, the call-out request is not cancelled. This allows for the response to return a cache header which can be used in later slot requests.

Single Request Architecture (SRA) Support

There is no SRA support at this time. In the event that SRA and RTC are attempted to be used on the same page, RTC will take precedence, and SRA will not be used.

Example

CoolSportsPublisher.com wants to use RTC to make call-outs to VendorFooBar, as well as to their own targeting endpoint, and they only want to allow for a 500ms timeout.

VendorFooBar is registered in callout-vendors.js as seen here:

/** callout-vendors.js */
vendors: {
...
   “vendorFooBar”: “https://vendorFooBar.com/foo?interest=INTEREST_NAME&color=COLOR_ID”,
...
};

CoolSportsPublisher’s targeting server endpoint is located at https://www.CSPAnalytics.com/tgt/f.html

With these requirements, CoolSportsPublisher should set up their amp-ad as follows:

/**sports.html */
...
<amp-ad type=doubleclick rtc-config=”{
    "vendors": {
       “vendorFooBar”: {
          “INTEREST_NAME”: “sports”
          “COLOR_ID”: “BLU”
       }
    },
    “urls”:   [“https://www.CSPAnalytics.com/tgt/f.html?page_id=1a2b3c”],
    “timeoutMillis”: 500
}”...></amp-ad>
...

Two parallel call-outs will result from this specification, the two urls called out to will be:
https://www.CSPAnalytics.com/tgt/f.html?page_id=1a2b3c
https://vendorFooBar.com/foo?interest=sports&color=BLU

Any call out that returns a response in less than 500ms will have its response merged into the DoubleClick ad request prior to sending.

[jasti]

CC @ampproject/ads @ampproject/a4a

[Aerlinger]

@bradfrizzell I'm assuming this implementation obviates the need for page-level configuration via a metatag as discussed in #8551?

The syntax proposed there was:

<meta name="amp-ad-remote-config-url" content="https://foo.your-domain.com/realtimeconfig.get?a=b&c=ATTR(data-slot)&url=CANONICAL_URL">.

[jasti]

@bradfrizzell We should probably close the other issue.

[bradfrizzell]

Correct, closed the other issue. A full implementation guide for publishers, ad networks, and third-party vendors is forthcoming.

[ampprojectbot]

This is a high priority issue but it hasn't been updated in awhile. @bradfrizzell Do you have any updates?

[bradfrizzell]

finished implementation here: #11377