Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debug tool like Professor Prebid for Protected Audience Auctions #937

Open
pm-harshad-mane opened this issue Dec 1, 2023 · 20 comments
Open

Comments

@pm-harshad-mane
Copy link

Multiple advertisement slots exist on the page where ad auctions occur among various participants. This auction setup is quite intricate, involving multiple players bringing diverse signals. At times, debugging the setup becomes necessary to gain a precise understanding of the auction's proceedings. The Prebid ecosystem, having evolved significantly, offers numerous methods and tools for bugging auction setups. Prebid auctions provide a range of event handlers to disseminate information about these events. List of the available events is maintained here, https://docs.prebid.org/dev-docs/publisher-api-reference/getEvents.html

Protected audience auctions prioritize privacy at their core. Consequently, individual participants are unable to access data points of other participants. Unlike analytics partners within Prebid, no player can join the auction solely to collect data that would help consolidate auction-related data points.

More details about the Prebid Chrome extension, Professor Prebid, can be found here: https://docs.prebid.org/tools/professor-prebid.html
Following are some screenshots of the open-source debug tool developed by the Prebid community:

Configuration of each Ad unit / Ad Slot:
image

Details of the bid received:
image

Timelines of the bid response:
image

The tool also creates an overlay on the actual ad-slot on the page: image

Considering the privacy-centric nature of the protected audience auction, no external script on the page can gather all the internal details of the protected audience auction so it would be helpful for the community if Google Chrome team develops some tool to display all such details in the Chrome Dev Tools. We will be happy to contribute for the development efforts.

Due to the privacy-focused design of the protected audience auction, external scripts present on the page are unable to access comprehensive internal details of these auctions. It would greatly benefit the community if the Google Chrome team could create a tool within Chrome Dev Tools to showcase these otherwise inaccessible details. We're eager to contribute to the development efforts required for this initiative. This tool would serve as a valuable resource, offering visibility into the intricate workings of protected audience auctions without compromising their privacy-centric structure.

Following is a list of features we would like the tool to have:

  • Display the ad-slot level details.
  • List of participants: component sellers, buyers, interest groups with respective configs
  • Details of the real-time signals: trusted bidding signals, trusted scoring signals
  • Auction configs and other signals
  • Received bids per seller with renderingURL, ad-metadata, k-anonymity score.
  • List the network calls separately apart from the usual Network Tab
  • Timeline of events
@JensenPaul
Copy link
Collaborator

A few followups on this issue:

  1. We discussed this for a while on the WICG Protected Audience call last week, notes here
  2. There was an ask about Chrome extensions talking to DevTools. The answer is yes, this is possible, see here
  3. We're hoping to add a better runAdAuction() integration with the DevTools -> Application -> Storage tab soon.

@dmdabbs
Copy link
Contributor

dmdabbs commented Feb 26, 2024

Hello @pm-harshad-mane. See Kevin Lee's PR for example DevTools protocol code to read PAAPI info:
https://github.com/privacysandbox/privacy-sandbox-demos/pull/269

Related Chromium issue
https://issues.chromium.org/issues/41489627

@dmdabbs
Copy link
Contributor

dmdabbs commented Feb 28, 2024

DevTools code moved here:
GoogleChromeLabs/trust-safety-demo#56

@pm-harshad-mane
Copy link
Author

Thanks @dmdabbs!

@pm-harshad-mane
Copy link
Author

Debug Information mentioned in #430 can be useful to be presented in this debug UI.

@morlovich
Copy link
Collaborator

So this landed in https://github.com/GoogleChromeLabs/trust-safety-demo/tree/main/protected-audience/samples/devtools-extension/src now, with Chrome side available fully since 123.0.6274.0

The example should be usable as-is (with a very "programmer UI" definition of usable), but it's basically meant to be sample code that anyone can use to help them get started if they want to build this sort of thing.
(It's Apache 2.0 licensed).

@pm-harshad-mane
Copy link
Author

Thanks @morlovich

@pm-harshad-mane
Copy link
Author

@morlovich, how can we view the inputs and outputs for each generateBid / scoreAd function call? Additionally, we're interested in viewing the responses from the KV servers, which serve as input to the aforementioned functions.

@morlovich
Copy link
Collaborator

Hmm, good questions. You can /almost/ get the inputs by using the special breakpoints, but the timing is wrong; it should be easy to add new ones right before running of the function rather than top-level, though might also need to do the "debug(generateBid)" thing at that point to get it exactly right. I will try this out and get back to you.
Also beware cumulative timeouts, and it is likely to slow things down.

Not sure if there is an easy way of seeing the return value, though the bid events have the important stuff; I could probably also add some score events?

I am not sure if any of the Network.* and Fetch.* mechanisms for intercepting things will work, I suspect not. We could probably hook them up. though I will need to check whether it makes it easier for people to do nefarious things. Just fetching the URL again will almost always give you the right stuff... except if it changes :(

@morlovich
Copy link
Collaborator

For first one, https://chromium-review.googlesource.com/c/chromium/src/+/5387607/1/content/services/auction_worklet/bidder_worklet.cc has the needed Chrome change, and sample modification to the extension stuck in a comment.

This produces something like this:

### Response:{"result":{"type":"object","value":{"0":{"ads":[{"metadata":{"adName":"default-ad"},"renderURL":"https://protected-audience-demo-dsp.web.app/ads/default-ad.html","renderUrl":"https://protected-audience-demo-dsp.web.app/ads/default-ad.html"}],"biddingLogicURL":"https://protected-audience-demo-dsp.web.app/bid.js","biddingLogicUrl":"https://protected-audience-demo-dsp.web.app/bid.js","enableBiddingSignalsPrioritization":false,"executionMode":"compatibility","name":"tv","owner":"https://protected-audience-demo-dsp.web.app","trustedBiddingSignalsSlotSizeMode":"none","useBiddingSignalsPrioritization":false},"1":{"isControversial":true},"2":{"windowInnerHeight":1014},"3":null,"4":{"adComponentsLimit":40,"bidCount":12,"forDebuggingOnlyInCooldownOrLockout":false,
(truncated)

on one of the demo sites.

There is probably a way of getting the return value too with some strategic stepping, since Debugger.paused has a 'returnValue' field in CDP inside the callframe structure:
https://chromedevtools.github.io/devtools-protocol/tot/Debugger/#type-CallFrame
... though that seems a little messy to fit into house the thing is structured.

@jlquaccia
Copy link

jlquaccia commented Mar 28, 2024

Hi @morlovich, thanks for the response! I tested out the code changes from https://chromium-review.googlesource.com/c/chromium/src/+/5387607/1/content/services/auction_worklet/bidder_worklet.cc locally on my end, however I am unable to access the code inside of this if statement during PA auctions (as I can't seem to get the if statement criteria to be met):

if ( method === 'Debugger.paused' && source.targetId && params.data?.eventName === 'instrumentation:beforeBidderWorkletGenerateBid' ) { let targetId = source.targetId; let childDebuggee = {targetId: targetId}; let frameId = params.callFrames[0].callFrameId; chrome.debugger.sendCommand( childDebuggee, 'Debugger.evaluateOnCallFrame', { callFrameId: frameId, expression: 'arguments', returnByValue: true, }, (response) => { console.log('### Response:' + JSON.stringify(response)); chrome.debugger.sendCommand(childDebuggee, 'Debugger.resume', {}); } ); }

This functionality should be available in the latest version of Chrome right? I have Version 123.0.6312.86 (Official Build) (arm64)

Seems the "Debugger.paused" event isn't getting triggered after setting the beforeBidderWorkletGenerateBid event breakpoint?

@morlovich
Copy link
Collaborator

It requires that CL, so it's not in any released version, sorry.

@jlquaccia
Copy link

@morlovich ok, thanks for the heads up 👍

@jlquaccia
Copy link

Hey @morlovich just a quick follow up question, what does "CL" refer to?

@morlovich
Copy link
Collaborator

Changelist. Roughly the same as what github would call "pull requests", just without an underlying pull operation involved (I think it's probably originally Perforce terminology, but I am not sure).

@pm-harshad-mane
Copy link
Author

@morlovich does that mean the change is yet to be released? If so, what is the target release schedule?

@morlovich
Copy link
Collaborator

It's not landed; I first have to make sure that my colleagues more familiar with ad industry feel it's not likely to be used to do nefarious things.

@pm-harshad-mane
Copy link
Author

Thanks for the update @morlovich !!

@pm-harshad-mane
Copy link
Author

pm-harshad-mane commented Apr 4, 2024

@morlovich until you get feedback from your colleagues, just to limit the access to the API... can you make this API available only for code executing through Chrome Extension ?

@jlquaccia
Copy link

jlquaccia commented Apr 9, 2024

Hi @morlovich, one question we have been curious about lately was around detecting when PA auctions end. I know there are win and topLevelBid events that can be listened for during an auction, but since these events don't always occur, is there a way to listen for when a PA auction has ended?

Also, just wanted to follow up on Harshad's question here as well about the code to view the inputs and outputs for each generateBid / scoreAd function call

@morlovich until you get feedback from your colleagues, just to limit the access to the API... can you make this API available only for code executing through Chrome Extension ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants