I2I: Amp Ad Real-Time Config

[keithwrightbos]

Publisher pages hosted by the CDN do not allow the publisher to specify different amp-ad implementations by user, device, etc. For amp-ad elements, remote HTML addresses this deficiency by allowing publishers to specify the frame in which Delayed Fetch executes, in which custom JavaScript dynamically modifies inputs to the ad tag. The primary use case is access to publisher party cookies to allow user-specific targeting. However, Fast Fetch does not allow custom JavaScript execution and thus is not executed (falls back to Delayed Fetch) when remote HTML meta tag is present. Solution proposed is for the publisher to set a new meta tag whose value is a URL (known henceforth as real-time config request) to be executed via GET XHR CORS. This not only ensures Fast Fetch supports real-time configuration, but will also allow for the request to be made earlier in the Delayed Fetch flow, as currently the 3p frame is attached as part of layoutCallback and therefore subjected to the 3 viewport (by default) and other throttling limitations.

Proposed meta tag with example URL:

<meta name="amp-ad-remote-config-url" content="https://foo.your-domain.com/realtimeconfig.get?a=b&c=ATTR(data-slot)&url=CANONICAL_URL">

The real-time config url will include cookies (so long as the CORS response headers indicate they are allowed via Access-Control-Allow-Credentials=true) and will be expanded via url replacement service for the following items (this may be extended as needed). Note that the length of the url will be capped at 4k characters via explicit truncation (allowing for the last parameter to be incomplete).

• ATTR(/name/) allowing for inclusion of attribute values from the amp-ad element
• RANDOM
• CANONICAL_URL the URL of the page as served from the publisher page
• SLOT_WIDTH & SLOT_HEIGHT which are the calculated height/width of the slot to be used for responsive layouts

For Delayed Fetch, the request will be made as soon as onLayoutMeasure executes, the slot is not in a prerender state, and prior to default 3p frame attachment. For Fast Fetch, the request will be made prior to building the ad URL (part of onLayoutMeasure, when slot is not in prerender state).

The AMP XHR service will include on the XHR request a __amp_source_origin parameter with value equal to domain of publisher page. This is because the standard XHR CORS Origin header value will be the CDN (still expects standard Access-Control-Allow-Origin response header whose value is the Origin request header). If access is allowed, the AMP runtime expects the response to include a header named AMP-Access-Control-Allow-Source-Origin whose value is equal to the __amp_source_origin value given in the request. Because this is a non-standard header, the response will also need to include the Access-Control-Expose-Headers header which is a comma separated list of custom header names and should therefore include AMP-Access-Control-Allow-Source-Origin.

The response must be JSON (expects Content-Type header equal to 'application/json') whose keys match attributes to be set on the amp-ad element and whose values are the attribute values to be set. These will then be used by the Fast Fetch implementations to build the ad request and other behavior. The following attributes are not allowed to be modified, and if provided in the response, will be ignored:

• Attribute keys that start with “on” to ensure no JavaScript execution
• style, class, height, width, type, layout

A timeout TBD will be imposed on the real-time config request such that if exceeded (or XHR returns non-200 or otherwise invalid response), the existing amp-ad configuration will be used. This could be configurable by the publisher such that timeout/error could allow for no ad request and collapse, but this configurability will not be included in the first version. Where possible, it is recommended that the response be cacheable by user in order to reduce network fetch delay. Client-side redirects will not be allowed.

Migration Plan

Initially, the remote HTML meta tag will still be honored as publishers move to the new implementation. If both are specified, real-time config will be used and Fast Fetch employed where supported. In the near future, remote HTML support will no longer be honored such that if present, no real time targeting will be made (Delayed Fetch will not use remote HTML path specified) and Fast Fetch will be executed where possible.

Example

<meta name="amp-ad-remote-config-url" content="https://foo.your-domain.com/realtimeconfig.get?a=b&c=ATTR(data-slot)&url=CANONICAL_URL">

Which has amp-ad element:

<amp-ad height=100 width=320 type="doubleclick" data-slot="/1234/a/b/c" data-some-attr="bar"></amp-ad>

Resulting real-time config url (assuming pub page is foo.your-domain.com/blah?foo=bar):

https://foo.your-domain.com/realtimeconfig.get?a=b&c=%2F1234%2Fa%2Fb%2Fc&url=foo.your-domain.com%2Fblah%3Ffoo%3Dbar&__amp_source_origin=foo.your-domain.com

Publisher server makes any potential call-outs to 3rd party vendors to build audience or other targeting to be set and are included in the response.

Example response (note inclusion of invalid attribute names):

{ “json”: {“a”:”b”, “c:123”, e:[1,2,3,4], “data-multi-size-validation”: “123x456,789x123”, “type”: “adsense”, “data-some-attr”: “hello” }

Resulting amp-ad element:

<amp-ad height=100 width=320 type="doubleclick" data-slot="/1234/a/b/c" data-multi-size-validation="123x456,789x123" data-some-attr="hello"></amp-ad>

[keithwrightbos]

Note that after looking at validation spec, plan is to move to a whitelist approach for valid attributes to ensure conformance. Any data- prefixed attributes would be allowed.

cc/ @Gregable

[jasti]

@ampproject/ads FYI

[michaelkleber]

One alternative that we discussed: Rather than the JSON response having a bunch of key-value pairs that turn into properties of the amp-ad tag, we could let the tag's values in the markup contain a new type of macro token to expand, and have the RTC call-out list all of the macros, and the response JSON have values for them. That is,

<meta name="amp-ad-remote-config-url" content="https://foo.your-domain.com/realtimeconfig.get?a=b&url=CANONICAL_URL">
...
<amp-ad height=100 width=320 type="doubleclick" data-slot="/1234/a/b/c" data-my-ads-targeting-attribute="{topic:'food',age:'{RTC:DEMOG_AGE}',sub:'{RTC:IS_SUBSCRIBER}'}></amp-ad>

Then the RTC could look through the amp-ad tags on the page and find all the {RTC:...} macros, issue a request containing a list of them, get a JSON response that says {DEMOG_AGE: '27', IS_SUBSCRIBER: 'true'}, and substitute those into the data-my-ads-targeting-attribute value seen by the amp-ad implementation.

This is less flexible in that you can't use it to, say, add new attributes to the markup. But it makes it easier to stitch together new and old information; the root I2I proposal where XHR responses replace in-markup values makes that use case to some server-side gymnastics.

We would be happy to hear opinions on the relative appeal of the ideas.

[cramforce]

Would it make sense to make this "batched" by default (only ever make a single request for the entire page)?

[michaelkleber]

In the original proposal, batching would mean applying the same attributes to each amp-ad slot on the page, or else making up some sort of unique identifier and a way to bundle/unbundle the request/response data.

In the alternative where the response contains values to substitute into keys present in the amp-ad attributes in the markup, one request for the whole page seems reasonable; we're solving the bundling/unbundling problem here by dumping it back on the page author, who effectively needs to namespace the keys across the different slots, if they want to be able to tell them apart.

[keithwrightbos]

My biggest concern is about how we will bundle config for multiple slots into a single request. I propose the following:

• Meta tag only provides end point with url replacement support for RANDOM and CANONICAL_URL
• An XHR CORS POST (to ensure no issues with url length) is sent to that end point whose body contains separate lines for each slot with the following information: the name/value of all attributes on the slot (minus "internal" attributes such as style, class) and the list of macros eligible for expansion via the response. Duplicate data across the slots should not be a concern given the request will be gzipped by the client (likewise for the response)
• The response will be a JSON array whose length equals the number of slots on the page where each entry is a JSON Object whose keys are the macros to replace and values are the value to use

The body of the POST will be inflated via the first amp-ad element's onLayoutMeasure + not in prerender state. It will determine the existence and values of other attributes on the page via document.querySelectorAll. Upon response it will update the attributes of the slots accordingly.

Edge conditions to consider:

• given querySelector will be used to "discover" other slots on the page, the extensions owning those slots will not have had a chance to execute and modify, add, or remove attributes. In addition, the measured size of the slots will not be available, only the value of height/width attributes. @lannka / @dvoytenko curious is server-side layout would make determining the measured size possible without waiting for onLayoutMeasure to execute.
• attributes included in post could themselves contain the macros. Publisher will need to keep this in mind and we will not support multiple replacement passes
• slots that have no macros will not be included in the config request such that if publisher includes meta tag but NO slots have a macro, no request will be made
• no urlReplacement service macros (e.g. performance API) will be supported as part of attribute expansion
• if the length of the JSON array returned != the number of slots, we should consider how to handle. Perhaps abort completely and collapse slots?
• auto-amp-ads will need to indicate when the DOM has been modified to include all slots cc/ @tlong2
• Are there other ways amp-ad elements can be dynamically added to the page (e.g. carousel)? If so, propose that slots included in previous config request have an attribute set (e.g. rtc-sent="true") so that they can be excluded and upon encountering slot without, it could initiate another config request. This should handle any concerns about dynamically added slots

[michaelkleber]

@keithwrightbos That all sounds good to me. I would suggest being more tolerant to response errors (, timeouts, etc): maybe a complaint in the JS console, but still execute all the amp-ad elements, leaving the macros unchanged.

[cramforce]

SSR has no impact on measurability of sizes. CORS POST with very declarative JSON sounds very good to me.