Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

POC: Adzerk Fast Fetch with template based preferential render #12765

Merged
merged 32 commits into from Jan 24, 2018
Merged

Conversation

glevitzky
Copy link
Contributor

Proof of concept in order to determine feasibility of template based AMP creatives for adzerk. AMP cache portions TBD.

AMP ad Fast Fetch implementation for adzerk network allowing for preferential render via mustache template. Template must have previously been pushed to AMP cache (yet to be implemented) in order to guarantee it is A4A valid.

Items to be addressed:

  • Consult with Adzerk to determine expected publisher tag implementation and resulting ad request url. Currently assumes single attribute named "src" whose format matches: https://adzerk.com\?id=#
  • Does not yet allow for automatic creation of amp-pixel and amp-analytics from known macro template values
  • sanitizeHtml strips out all <script> tags within body causing extensions that utilize script for configuration to not work properly (e.g. amp-animations & amp-analytics)
  • Does not leverage template service. This was intentional design decision but could be altered.
  • AMP template is "consumed" by the client for preferential render using DOMParser in order to determine fonts/extensions and remove unnecessary portions of head (scripts, boilerplate). This could be moved server side as part of AMP cache integration.
  • Additional test coverage.

Original PR: #12616

/cc @ampproject/a4a

}

AMP.extension('amp-ad-template', '0.1', AMP => {
AMP.registerElement('amp-ad-template', AmpAdTemplate);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should not be an extension. it can be just another file in amp-a4a/0.1/ folder for code sharing

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

/** @const {string} */
const TAG = 'amp-ad-template';

export class AmpAdTemplate {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Naming: AmpAdTemplates since this is more like a template manager class

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

/**
* Fetch and parse template from AMP cache. Result is stored in global in
* order to reduce overhead when template is used multiple times.
* @param {number} templateId
Copy link
Contributor

@lannka lannka Jan 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as discussed, let's use a templateUrl string.

For example,

Template canonical URL: https://adserver.com/amp_template_1
=>
Template proxy URL https://adserver-com.cdn.ampproject.org/a/s/adserver.com/amp_template_1

The domain name in the proxy URL serves as namespace and the path serves as an ID.
It avoids all the hassles for ad network to submit templates and manage IDs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

* @param {!Element} element Parent element containing template.
* @param {!Window=} opt_window
*/
populateTemplate(templateValues, element, opt_window) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

naming nitpick: render()

can we merge element and opt_window?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

import {dev} from '../../../src/log';
import {getMode} from '../../../src/mode';
import {getAmpdoc} from '../../../src/service';
import {Bind} from '../../amp-bind/0.1/bind-impl';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will compile the whole amp-bind extension into this extension

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can I avoid doing so?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might want to get the binder from service.

@choumx correct me if I'm wrong
try: adoptServiceForEmbedIfEmbeddable()

Copy link
Contributor Author

@glevitzky glevitzky Jan 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't think this is the way. I get the following error:

Error: Service amp-bind not found on parent or doesn't implement EmbeddableService.

Even if this did work, I'd still need some way to call setState().

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should retrieve services from the corresponding function in src/services.js --
in this case, bindForDocOrNull.

* @param {number} templateId
* @return {!CachedTemplateDef}
*/
retrieveTemplate(templateId) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

naming nitpick: fetch(templateUrl)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

};
const cacheKeys = /**@type {!Array<number>}*/
(Object.keys(this.templateCache_));
if (cacheKeys.length > 5) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe worth to implement a separate helper class: LcuCache?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

this.ampCreativeJson_ = null;

ampAdTemplate = ampAdTemplate ||
new AmpAdTemplate(this.win, this.parseTemplate_.bind(this));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of passing in parseTemplate function in constructor, can you do:

AmpAdTemplates.fetch(url).then(t => {
  this.parseTemplate_(t);
});

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


class LRUCache {
/** @param {number} capacity */
constructor(capacity) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can pass in a generator function in constructor. Then we only need one public method for the LRUCache: get()

Also, worth to extract the code into src/utils/cache.js for sharing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really sure how a generator would fit in here. Can you give an example of how you imagine the cache being used? If it's not too strong a preference on your end, I'd prefer to keep it with get/put.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

class LRUCache {
  constructor(capacity, generator);

  get(id) {
    ...

    if (!this.cache_[id]) {
      this.cache_[id] = generator(id);
    }
    return this.cache_[id];    
  }
}

something like this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I'm still not understanding. Who calls next(), and how does this eliminate the need for put()?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is holding up release. Let's change this in a future PR. I don't think anyone will be using this code for a while anyway.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will be used like:

constructor() {
  this.cache_ = new LRUCache(5, url => {
      return Services.xhrFor(this.win_).fetchText(url);
  });
}
...

fetch(url) {
  return this.cache_.get(url);
}

there are other more important comments not addressed in this PR, so realistically, i don't think it should hold the release.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot to mark some of the comments as done. Will address the others shortly.

@lannka
Copy link
Contributor

lannka commented Jan 23, 2018

There is a couple of remaining comments unaddressed. Please reply to each of them when done.

@amphtml-team
Copy link

amphtml-team commented Jan 24, 2018 via email

@amphtml-team
Copy link

amphtml-team commented Jan 24, 2018 via email

@glevitzky
Copy link
Contributor Author

glevitzky commented Jan 24, 2018 via email

Copy link
Contributor

@lannka lannka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes!

<div fallback></div>
</amp-ad>

<!--<amp-ad width=300 height=250
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uncomment them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

* amp-mustache. AMP creative response will consist of the following JSON
* object with two fields:
*
* - ampCreativeTemplateId: number value for template ID. Template must already
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we follow the naming in the design doc?

templateUrl,
data,

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

*/
getTemplateProxyUrl_(url) {
const loc = parseUrl(url);
const hostClean = startsWith(loc.host, 'www.')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to remove www..

const hostClean = startsWith(loc.host, 'www.')
? loc.host.slice(4)
: loc.host;
return loc.protocol + '//' + hostClean.replace('.', '-') + '.' +
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please refer to AMP cache URL format

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably won't fix until morning, but besides removing 'www', what's wrong with this format?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe not clear from the code, but this converts https://adserver.com/amp_template_1 to https://adserver-com.cdn.ampproject.org/a/s/adserver.com/amp_template_1 as mentioned.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Converting the AMP document domain from IDN (Punycode) to UTF-8.
  • Replacing every "-" (dash) with "--"(2 dashes).
  • Replacing every "." (dot) with a "-" (dash).
  • Converting back to IDN (Punycode).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

* string value used to dynamically update the template
*
* Additionally, ad response must include header indicating AMP creative
* template response: AMP-template-amp-creative: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of having the header value to be true, we can put amp-mustache, to differentiate templating languages.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

*/
getTemplateProxyUrl_(url) {
const loc = parseUrl(url);
return loc.protocol + '//' +
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

proxy URL is always https

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

checkStillCurrent();
this.ampCreativeJson_ = /** @type {!AmpTemplateCreativeDef} */
(tryParseJson(body) || {});
const proxyUrl = getMode(this.win).localDev
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you move the proxy URL logic into AmpAdTemplates for sharing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

const loc = parseUrl(url);
return loc.protocol + '//' +
loc.hostname.replace(/-/g, '--').replace(/\./g, '-') +
'.' + urls.cdn.slice(8) + '/a/s/' + loc.hostname + loc.pathname;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's use /c/s/ for now, before AMP cache supports /a/s/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Contributor

@lannka lannka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. good to go, thanks for the work!

/**
* Fetch and parse template from AMP cache. Result is stored in global in
* order to reduce overhead when template is used multiple times.
* @param {string} templateUrl CDN Proxy URL to template.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

canonical URL

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

`0.1/data/${match[2]}.template`;
fs.readFileAsync(filePath).then(file => {
res.setHeader('Content-Type', 'application/json');
res.setHeader('AMP-template-amp-creative', 'true');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

amp-mustache

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

pc.cwd() + '/extensions/amp-ad-network-adzerk-impl/0.1/data/' + match[1];
fs.readFileAsync(filePath).then(file => {
res.setHeader('Content-Type', 'application/json');
res.setHeader('AMP-template-amp-creative', 'true');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

amp-mustache

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@lannka lannka merged commit e3f6d9c into master Jan 24, 2018
@lannka lannka deleted the adzerk_a4a branch January 24, 2018 20:46
gzgogo pushed a commit to gzgogo/amphtml that referenced this pull request Jan 26, 2018
…oject#12765)

* POC: Adzerk Fast Fetch with template based preferential render

* fix dep check

* fix test failures

* Using templating service.

* yarn.lock revert

* yarn.lock

* yarn.lock

* Attempt ampproject#4 at yarn fix...

* Test fix.

* Update mock templates to use <template>s.

* Reverting mode-object.

* Yarrrrn

* Moving toward amp-bind.

* amp-bind integration.

* Removing the need for amp-state.

* Revert "Removing the need for amp-state."

This reverts commit 2134f8c.

* Modifying transformed template.

* Factored our ad templating into new extension.

* Deleting test file.

* Lint fixes and other minutiea.

* Added comments.

* PR feedback.

* Removed unused function.

* Added tests for templates and cache.

* Adding tests.

* presubmit fixes.

* Reverting to amp-mustache.

* Url transforming, uncommenting, and other feedback.

* Removing duplicate files.

* Moving url transform to amp-ad-templates.

* Test fixes

* header values + minor comment change.
@lannka
Copy link
Contributor

lannka commented Jan 31, 2018

For #12903

RanAbram pushed a commit to RanAbram/amphtml that referenced this pull request Mar 12, 2018
…oject#12765)

* POC: Adzerk Fast Fetch with template based preferential render

* fix dep check

* fix test failures

* Using templating service.

* yarn.lock revert

* yarn.lock

* yarn.lock

* Attempt #4 at yarn fix...

* Test fix.

* Update mock templates to use <template>s.

* Reverting mode-object.

* Yarrrrn

* Moving toward amp-bind.

* amp-bind integration.

* Removing the need for amp-state.

* Revert "Removing the need for amp-state."

This reverts commit 2134f8c.

* Modifying transformed template.

* Factored our ad templating into new extension.

* Deleting test file.

* Lint fixes and other minutiea.

* Added comments.

* PR feedback.

* Removed unused function.

* Added tests for templates and cache.

* Adding tests.

* presubmit fixes.

* Reverting to amp-mustache.

* Url transforming, uncommenting, and other feedback.

* Removing duplicate files.

* Moving url transform to amp-ad-templates.

* Test fixes

* header values + minor comment change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants