Implementing prebid-cache to support Video ads #1277

dbemiller · 2017-06-07T17:18:04Z

PR which leverages prebid-server as a video ad cache. The goal here is to give publishers an ID which they can send to the ad server as a query param.

Per the feedback, I'm not exposing the callback to users anymore. The bidmanager simply waits to add the bid until the call to the cache succeeds.

protonate · 2017-06-08T16:58:43Z

src/bidmanager.js

+ * cache ID has been returned.
+ */
+function prepVideoBid(bid) {
+  // Queue callbacks which arrive before the cache has returned.


We use queues of functions that are to be called when some async operation completes, including pbjs.cmd, events queue and the in-progress renderers PR for callbacks to fire once the renderer is loaded. It would be good to generalize this with a common module.

This got deleted anyway... it's not necessary if we don't expose the callback to the user.

protonate

This is looking good, I've a bunch of notes, many focused on the heavy lift of what we name things, and some callouts for later bikeshedding, but the big note would be the object.adServer and API fluctuations if more than one ad server.

protonate · 2017-06-26T20:08:07Z

src/adServerManager.js

+ * through the properties and functions on `pbjs.adserver`.
+ *
+ * If publishers bundle with more than one Ad Server, they'll need to interact with them through
+ * `pbjs.adservers.{adServerCode}`, where the adServerCode is defined by the ad server itself.


It would be good to keep the API consistent if the ad server to use could be determined through config or from the Bid object, Ad Unit, etc.

Can it be? I'm all for deriving properties and avoiding data redundancies.

Which properties on the Bid or AdUnit could be used to derive it?

Also beware: that would prevent ad server functionality from being a module, since we wouldn't know which ad server was needed until runtime. Do you think that's a worthwhile tradeoff?

We need to introduce a new config object I think.

I'd imagine the pub would be configured with an Ad Server definition (including "none") and an AdUnit could be configured with an override to use, if present.

Chatted about this outside of the PR. I removed pbjs.adserver, so the API will always be pbjs.adServers.dfp.buildVideoUrl(options).

protonate · 2017-06-26T20:59:11Z

src/adServerManager.js

+/**
+ * @typedef {Object} VideoSupport
+ *
+ * @property {string} code The identifying code for this adserver. Note that this contributes to


I know this is used throughout Prebid.js but would love to find another word to use in place of code. name, perhaps, and start migrating to that.

Done. I do like name better than code... and this is a pretty good place to start using it, since the API is pretty internal.

protonate · 2017-06-26T21:02:42Z

src/adServerManager.js

+ * If this is the only code we've seen, attach functionality to $$PREBID_GLOBAL$$.adServer.
+ * If we've seen any other codes, attach functionality to $$PREBID_GLOBAL$$.adServers[code].
+ *
+ * @param {object} object The object where this function should manage namespaces.


This could be called adServers by getting rid of object.adServer default, thereby promoting object.adServers to just adServers. Then the API can be generic functions and use a strategy pattern to determine which ad server-specific functions to use, based on adUnit or page level config probably.

protonate · 2017-06-26T21:05:07Z

src/adServerManager.js

+ * @param {string} code The code for this ad server.
+ * @return {object} An object where functions for dealing with this ad server can be added.
+ */
+function prepareNamespaceIn(object, code) {


A strategy pattern would remove the need for much of the "default state management" code here.

I'm also unsure of the meaning of prepareNamespaceIn -- do you mean "in prebid with this code", or "in bound", or ... ?

Not relevant anymore because I deleted it, but... it was supposed to read like "prepare a namespace in object for {code}".

I'm generally trying to name functions so that the code which calls them reads as closely to english as possible.

protonate · 2017-06-26T21:34:01Z

src/adServerManager.js

+ *   the user-facing API. For example, if the adserver code is 'dfp', and the Prebid bundle contains two
+ *   or more ad server modules, then publishers will access its functions through 'pbjs.adservers.dfp'.
+ *
+ * @method {VideoAdUrlBuilder} buildVideoAdUrl


I'd prefer @function rather than the alias/synonym, and consistency at the least.

yeah that's fair. Actually I agree, cause in my mind, method seems to signify something that shouldn't be ripped off of the object, because it's relying on this. function signifies something that can be passed around freely.

I'll change it.

protonate · 2017-06-27T14:17:29Z

src/videoCache.js

+/**
+ * Takes the user-defined callback, and wraps it into our ajax function's API.
+ *
+ * This also needs to work around the fact that the cache server returns a 200 on "data not found"...


We should change this behavior server-side?

The semantics are debatable...

As a server: someone just told you to return the data stored in the cache. If you looked in the cache and successfully found no value, did you complete the request successfully? What if someone told you to store "nothing", in one of their earlier put requests?

I answered that with a definite ¯_(ツ)_/¯, and decided to work with Sweeny's implementation.

A 204 would provide a definite "successful request with no content response".

I think my broader point is we also control the server and don't have to work around anything that doesn't make sense to us.

I'd buy that. So trying to be clear, my understanding is that you think the cache should:

If it doesn't recognize the key, return a 404.

If it recognizes the key and stores "nothing", return a 204.

If it recognizes the key and stores "something", return a 200 with the stored data.

Correct?

er... and one other judgment call which needs agreement:

If the videoCache can't find the video, it should be an error.

(I didn't see any discussion on this point... but it's an important one. If you agree, then a prebid-cache 204 "success" would still be considered an "error" from this file's point of view, and all this code would still be necessary anyway)

protonate · 2017-06-27T14:19:55Z

src/videoCache.js

+ * Fetch the VAST XML associated with cacheId, and execute the callback function with it.
+ *
+ * @param {string} cacheId The ID of the video whose content should be retrieved.
+ * @param {videoCacheRetrieveCallback} callback A callback function which should be run once the


We've been conforming to calling this a done callback.

Didn't know that... I'll update it

protonate · 2017-06-27T14:22:25Z

src/videoCache.js

+ * Make a server call for the given cacheId, and execute the callback function when they get back to us.
+ *
+ * @param {string} cacheId The ID of the video whose content should be retrieved.
+ * @param {videoCacheRetrieveCallback} callback A callback function which should be run once the


Call this done?

protonate · 2017-06-27T14:25:01Z

src/videoCache.js

+ * [cacheId, cacheValue] pairs locally so that we don't go to the network again if
+ * the publisher requests the same VAST content more than once.
+ */
+function memoize(retrievalFunction) {


Great, can you add some info on when this is used? For example, I think it would only be when a bid is returned to an auction from the (pending) BidPool. Is my understanding accurate?

It's used right below it, in the exported function :).

Per the docs, the goal here is to prevent multiple server calls.

If we successfully fetched a thing from the server, we can store it in page memory for the next time.

If the server fetch failed, we want to retry the call (might have just been a network problem).

It might have some more general uses, but... better private than public and only used once, imo. If refactoring, it's much easier to make more visible rather than less.

I don't think we need this functionality at all. We don't actually invoke the read from the cache, the video player does. Am I missing something?

You know... you're totally right.

Whenever I write something which interacts with an HTTP API, my first instinct is always "let's wrap that API in one from {my language} so that it's contained."

So.... I wrote it. Then wrote the video player integration. Didn't even notice that it was never called by real code.

Is deleted now.

protonate · 2017-06-27T14:27:21Z

test/spec/unit/bidmanager_spec.js

+     * Initialize the global state so that the auction-space looks like we want it to.
+     *
+     * @param {Array} adUnits The array of ad units which should appear in this auction.
+     * @param {function} bidRequestTweaker A function which accepts a basic bidRequest, and


Best name yet!

mkendall07 · 2017-06-28T20:42:23Z

modules/dfpAdServerVideo.js

+
+  const derivedParams = {
+    correlator: Date.now(),
+    description_url: encodeURIComponent(bid.descriptionUrl),


Do we still need to send description_url? Should we be getting bid.vastUrl?

Per the discussion in chat, I removed this. Publishers can still send it in themselves.

I don't think the vastUrl is related... but it's not needed by this point in the code anyway. The URL has been sent to the cache, so it's the { hb_uuid: bid.videoCacheKey } line which adds that to the DFP URL.

mkendall07 · 2017-06-28T20:50:07Z

src/bidmanager.js

  }

-  if (bidsBackAll()) {
-    exports.executeCallback();
+  if (isValid()) {


I'd prefer to see this code block at the beginning of the addBidResponse method as it's easier to read the code that way, vs scanning and trying to see what block is not wrapped inside a function.

Makes sense. Is fixed (after the next push)

mkendall07 · 2017-06-28T20:59:37Z

src/videoCache.js

+
+const PUT_URL = 'https://prebid.adnxs.com/pbc/v1/put'
+const GET_URL = 'https://prebid.adnxs.com/pbc/v1/get?';
+const EMPTY_VAST_RESPONSE = '<VAST version="3.0"></VAST>'


empty response will need a few required params. Example: https://stash.corp.appnexus.com/projects/RAD/repos/app_prebid-video-cache/browse/src/controllers/vast.js#6
Spec: https://github.com/InteractiveAdvertisingBureau/vast/blob/master/vast3_draft.xsd#L21

Looks like this disappears anyway. It was only being used in in store, which turned out to be unnecessary.

mkendall07 · 2017-06-28T21:00:16Z

src/videoCache.js

+  // Technically, this is vulnerable to cross-script injection by sketchy vastUrl bids.
+  // We could make sure it's a valid URI... but since we're loading VAST XML from the
+  // URL they provide anyway, that's probably not a big deal.
+  return '<VAST version="2.0"><Ad id=""><Wrapper><AdSystem>prebid.org wrapper</AdSystem><VASTAdTagURI><![CDATA[' +


Are we supporting VAST 2 or VAST3?

Per the chat, it doesn't sound like it matters. I'll bump it up to 3.0, since they're more likely to maintain backwards compatibility with that version if the standard ever moves forward any more.

mkendall07 · 2017-06-28T21:00:49Z

src/videoCache.js

+  // Technically, this is vulnerable to cross-script injection by sketchy vastUrl bids.
+  // We could make sure it's a valid URI... but since we're loading VAST XML from the
+  // URL they provide anyway, that's probably not a big deal.
+  return '<VAST version="2.0"><Ad id=""><Wrapper><AdSystem>prebid.org wrapper</AdSystem><VASTAdTagURI><![CDATA[' +


Prefer string templates instead of concatenation.

Makes sense

mkendall07 · 2017-06-28T21:03:37Z

src/videoCache.js

+      done(null, ids);
+    },
+    error: function(statusText, responseBody) {
+      done(new Error('Error storing video ad in the cache: ' + statusText + ': ' + JSON.stringify(responseBody)), []);


string templates again.

mkendall07 · 2017-06-28T21:04:44Z

src/videoCache.js

+ * @return {Function} A callback which interprets the cache server's responses, and makes up the right
+ *   arguments for our callback.
+ */
+function shimStorageCallback(done) {


Don't love the idea of calling a function a shim

I agree it's not the clearest, but... is there anything you like better? transform? wrap? Other?

I haven't come up with anything I really liked.

mkendall07 · 2017-06-28T21:07:44Z

src/videoCache.js

+ * [cacheId, cacheValue] pairs locally so that we don't go to the network again if
+ * the publisher requests the same VAST content more than once.
+ */
+function memoize(retrievalFunction) {


I don't think we need this functionality at all. We don't actually invoke the read from the cache, the video player does. Am I missing something?

…through the bid.

…hanged video support so that only bids which have been cached are valid.

…n be transformed into a module.

…stency with the normal conventions.

…e Ive messed it up multiple times myself already during testing, expecting it to have a different name.

…where.

protonate · 2017-07-10T19:42:22Z

integrationExamples/gpt/video_test.html

@@ -0,0 +1,132 @@
+


This can be moved to ./test as it isn't really an integration example.

mkendall07 · 2017-07-10T21:48:50Z

src/bidmanager.js

+  // Video bids may fail if the cache is down, or there's trouble on the network.
+  function tryAddVideoBid(bid) {
+    store([bid], function(error, cacheIds) {
+      doCallbacksIfNeeded();


Shouldn't we move this line after the bid is added to the auction? I realize that this is not consistent with the previous behavior, but at this point we should have a bid cached, so why not include it? @protonate @dbemiller

* Added code to cache video content, and made the cache key accessible through the bid. * Renamed the method we add to the bid. * Removed an extraneous function. * Removed some accidental console.log messages * Added unit tests for the video cache, and fixed a bug or two. * Refactored the bidmanager a bit to help wrap my head around it, and changed video support so that only bids which have been cached are valid. * Implemented video end-to-end. Waiting on the modules PR so that it can be transformed into a module. * Better fix to the utils bug. Modernized the imports. * Updated documentation. * Added a few unit tests, and added fixtures for shared data structures. * Added more thorough tests, and refactored a bit for better code reuse. * Renamed a function. * Added tests for adding video bids when the cache fails. * Removed an unused import. * Added tests for the adServerManager, and renamed some files for consistency with the normal conventions. * Renamed the dfpVideo module, and added unit tests for it. * Moved dfpAdServerVideo into the modules directory. * Fixed a bug, and added a regression test to catch it in the future. * Added a bare-bones example page. * Made some cosmetic changes to names and documentation. * Removed the shifty API. Updated unit tests. Renamed a property because Ive messed it up multiple times myself already during testing, expecting it to have a different name. * Most code review comments. Still need to look into the details of VAST. * Deleted some unused code, and upped the vast version to use 3.0 everywhere. * Made the cache use the new endpoints * Fixed style. * Moved the test page into the tests directory. * include the bid that ended the auction

mkendall07 added the in progress label Jun 7, 2017

dbemiller added the video label Jun 7, 2017

protonate self-requested a review June 7, 2017 17:36

protonate assigned protonate and dbemiller and unassigned protonate Jun 7, 2017

protonate reviewed Jun 8, 2017

View reviewed changes

dbemiller mentioned this pull request Jun 22, 2017

Fixing some fragile tests #1318

Merged

dbemiller force-pushed the feature/video-cache branch from d05eb4f to 15e6db4 Compare June 23, 2017 13:35

protonate suggested changes Jun 27, 2017

View reviewed changes

dbemiller force-pushed the feature/video-cache branch from 037803a to bf92a4c Compare June 27, 2017 15:23

mkendall07 reviewed Jun 28, 2017

View reviewed changes

dbemiller added 18 commits June 30, 2017 10:48

Added code to cache video content, and made the cache key accessible …

e5c7920

…through the bid.

Renamed the method we add to the bid.

5985f39

Removed an extraneous function.

65863a6

Removed some accidental console.log messages

219df5d

Added unit tests for the video cache, and fixed a bug or two.

73b0d26

Refactored the bidmanager a bit to help wrap my head around it, and c…

57139cc

…hanged video support so that only bids which have been cached are valid.

Implemented video end-to-end. Waiting on the modules PR so that it ca…

77831e1

…n be transformed into a module.

Better fix to the utils bug. Modernized the imports.

da3459d

Updated documentation.

3f94cfc

Added a few unit tests, and added fixtures for shared data structures.

008eda0

Added more thorough tests, and refactored a bit for better code reuse.

74ebd03

Renamed a function.

c20c6b1

Added tests for adding video bids when the cache fails.

ab4745d

Removed an unused import.

e3673aa

Added tests for the adServerManager, and renamed some files for consi…

17e77dc

…stency with the normal conventions.

Renamed the dfpVideo module, and added unit tests for it.

0fc1bd1

Moved dfpAdServerVideo into the modules directory.

285e0ca

Fixed a bug, and added a regression test to catch it in the future.

bfb1cc6

dbemiller added 5 commits June 30, 2017 10:48

Added a bare-bones example page.

fcf673a

Made some cosmetic changes to names and documentation.

09807f3

Removed the shifty API. Updated unit tests. Renamed a property becaus…

0ebe649

…e Ive messed it up multiple times myself already during testing, expecting it to have a different name.

Most code review comments. Still need to look into the details of VAST.

7df1e7e

Deleted some unused code, and upped the vast version to use 3.0 every…

4b9624b

…where.

dbemiller force-pushed the feature/video-cache branch from a4e7ff5 to 4b9624b Compare July 5, 2017 18:24

Made the cache use the new endpoints

863bfb4

dbemiller force-pushed the feature/video-cache branch from fadd38b to 863bfb4 Compare July 10, 2017 12:55

Fixed style.

72fd424

protonate reviewed Jul 10, 2017

View reviewed changes

protonate previously approved these changes Jul 10, 2017

View reviewed changes

Moved the test page into the tests directory.

fb4045b

dbemiller dismissed protonate’s stale review via fb4045b July 10, 2017 20:40

mkendall07 reviewed Jul 10, 2017

View reviewed changes

mkendall07 previously approved these changes Jul 10, 2017

View reviewed changes

protonate previously approved these changes Jul 11, 2017

View reviewed changes

include the bid that ended the auction

da637df

mkendall07 dismissed stale reviews from protonate and themself via da637df July 11, 2017 17:48

mkendall07 approved these changes Jul 11, 2017

View reviewed changes

mkendall07 merged commit 56bdd27 into master Jul 11, 2017

mkendall07 deleted the feature/video-cache branch July 11, 2017 18:14

This was referenced Jul 12, 2017

Fixing some recent bugs in addBidResponse #1372

Merged

Added MobFox Adapter #1312

Merged

dbemiller mentioned this pull request Jul 20, 2017

Missing parameters in bidAdjustment events #1398

Closed

Implementing prebid-cache to support Video ads #1277

Implementing prebid-cache to support Video ads #1277

Conversation

dbemiller commented Jun 7, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

protonate left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbemiller Jun 27, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

protonate Jun 27, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbemiller Jun 27, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbemiller Jul 5, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbemiller commented Jun 7, 2017 •

edited

dbemiller Jun 27, 2017 •

edited

protonate Jun 27, 2017 •

edited

dbemiller Jun 27, 2017 •

edited

dbemiller Jul 5, 2017 •

edited