Batch Requests to DSP Trusted Server

[davidhqr]

Introduction

Chrome currently sends a single GET request to the DSP trusted server with all the trusted bidding keys to query for to fetch real-time bidding signals. This can be problematic for AdTechs since GET requests are typically constrained to some rather-small size limit that varies across different web servers (example: the default URL length limit for Apache is 8.2KB).

Limitations of Current Approach

Under the current approach, a single GET request is sent to the DSP trusted server. Depending on what the AdTech chooses to use as the trusted bidding key, the request could easily overflow the GET request length limit and be dropped. Dropped requests will adversely affect the advertiser’s ability to provide bidding signals for the auction.

Example: If the IG name is chosen to be the trusted bidding key, then under the current FLEDGE implementation there could be up to 1000 IGs per request as keys. For a request with 1000 keys, each key can only be up to 8.2B in length if the GET request length limit is 8.2KB, not even taking into account the request URL and headers.

Proposed Changes

1: Split the DSP Trusted Server Request

We propose for the DSP trusted server request to be split up into several batches in order to accommodate GET request length limits. Instead of sending a single request, Chrome will split the keys up into a certain number of batches to send in parallel. Upon receiving the request from each batch, Chrome will join the responses from each request to construct the whole response.

2: Add New Field to Specify the Maximum URL Length

We also want AdTechs to be able to specify the maximum URL length of a request. For this, we propose to add a new field to the IG: maxTrustedBiddingUrlLength.

When preparing requests to the DSP trusted server, Chrome will group IG with the same trustedBiddingSignalsUrl together as before. Afterwards, Chrome will find the minimum maxTrustedBiddingUrlLength value among these IGs and batch them such that no batch has a URL length exceeding it.

Alternative Solution

An alternative solution is to change the request HTTP method from GET to QUERY, POST, or PUT. These HTTP methods have much looser length restrictions.

Summary of Asks

1. Split the DSP trusted server request into batches.
2. Add a new IG field to specify the maximum trusted bidding URL length.

[MattMenke2]

Don't know about other browsers, but Chrome doesn't cache POSTs (we'd need to cache and verify the POST body, which we don't currently do, and modifying the cache's indexing logic is something that's a rather major investment). If it weren't for that limitation, that would be my preferred approach.

So it does seem like splitting up requests is probably the best way to do this, unless we want to make a special cache for these responses that can coincidentally manage cached responses to POSTs (longer term, we'll need to make our caching better respect user privacy somehow, but that may just be using special network partitions or somesuch, which our HTTP cache already somewhat supports).

[JensenPaul]

This does sound like an area we could improve on. As Matt mentioned, an approach involving POST would involve substantially more work building an alternate caching mechanism. I also agree that splitting up the requests is probably the best approach. I think we should also consider fixing this problem for overly large trusted scoring signal URLs as well as they could hit these same limitations, perhaps even sooner as these URLs include other URLs (the creative render URL). I'd propose adding a new interest group field, maxTrustedBiddingSignalsURLLength, and a new auction config field, maxTrustedScoringSignalsURLLength, that each specify the maximum length of the corresponding URLs.

[davidhqr]

Good point about large scoring signal URLs as well. This solution sounds good to me.

[dmdabbs]

Is there a Monorail issue tracking this? I only found the URLlength CL.

[rdgordon-index]

#158 (comment)

[dmdabbs]

Answering my own question above:
https://issues.chromium.org/issues/324416012