forDebuggingOnly availability

[jonasz]

Hi,

I was wondering, what is the plan for the forDebuggingOnly reporting functions and their availability? Will they be supported during the mode a and mode b testing phases?

Best regards,
Jonasz

[ajvelasquezgoog]

Hi @jonasz, yes they are supported in both A and B modes.
forDebuggingOnly for loss reporting, the plan is to retire it by 3PCD
forDebuggingOnly for win reporting, the plan is to keep it around until at least 2026

[ajvelasquezgoog]

@jonasz we are actually thinking more about the privacy risks of the two parts of the forDebugOnly APIs, we need to think more about this, let us get back to you soon, hopefully next week

[fhoering]

Do you think some sampled mode could be acceptable in the long term ? Something small enough that it doesn't allow to do any user identification, like 1% of forDebuggingOnly.reportWin & forDebuggingOnly.reportLoss ?
Those flags are extremely useful to be able to debug client side code and allow e2e testing the full pipeline at event level. Not sure this could easily be reproduced with aggregated reporting of technical errors.

[jonasz]

@ajvelasquezgoog friendly ping, any updates on this issue?

[ajvelasquezgoog]

We thank everyone interested for your patience on getting updates on this matter. We have been working closely with and collecting feedback from stakeholders over the last several weeks, and examined the efforts required to adapt to the full removal of these functions by the 3PCD deadline.

The intent of forDebuggingOnly has always been since inception to be used for troubleshooting purposes as adtech builds and tests their integration with the Protected Audience API, with our current plan of record indicating that these functions will be deprecated at 3PCD.

The incremental feedback that we have been receiving in the last few months on this plan can be summarized as follows:

• Net new onboarding by adtech to our APIs will certainly continue after 3PCD, as more and more adtech solidify their integrations plans, even if they cannot get to them at exactly 3PCD.
• Once an integration is live, there are continued needs to have debugging abilities in the situations where operational monitoring and alerting that adtech has in place flags a potential production issue that has a (partial) dependency on the Protected Audience API. While the browser provides a comprehensive set of tooling to allow for bug reproduction and analysis, it remains a critical need for adtech to be able to access samples of debugging information to faster and more confidently do root cause analysis in escalation situations.
• A variation of this last use case that we have heard of that also depends on access to samples of this debugging information is in what we can best describe as proactive quality control audits, in which samples are taken that include inputs and outputs of the different worklets and evaluated/compared against internal benchmarks done in isolation in a laboratory environment

Given that, we think there is a path to continue supporting these use cases with a certain level of fidelity that will be acceptable, that also continues to meet our privacy goals. In essence we think it is possible to keep the forDebuggingOnly API and its signature as-is while significantly downsampling how often does a call to forDebuggingOnly actually fetch the report destination URL and significantly reduce the risk of bad actors establishing cross-site linkages or re-identifying users.

In essence the proposal entails the introduction of 3 Chrome-controlled variables that will modify the current behavior of the forDebuggingOnly.reportAdAuctionWin() and forDebuggingOnly.reportAdAuctionLoss() calls by 3PCD rollout. Please recall that these calls can be made by sellers in scoreAd() and for every buyer’s generateBid() function for each interest group that is participating in the auction.

New variable 1: Sampling Rate. Denotes how often a call to the forDebuggingOnly methods actually results in the report’s destination URL being fetched. Let us call this variable f, where how often the call happens is defined as 1 / f. Additionally, let’s define an outcome variable o as a binary [TRUE | FALSE], where o = TRUE one out of f times based on a randomizing function implemented by Chrome. We propose a value of 1000 for f.

New Variable 2: Cooldown Period. Denotes for how long (in days) should a single Chrome client, for a given calling adtech, return the same FALSE result for o after running the randomizing function that determines that the result should be FALSE. Let us call this variable c, and we propose a value of 365 days (1 year) for it.

New Variable 3: Lockout Period. Denotes for how long (in days) should a single Chrome client, for any and all calling adtech return a FALSE result for o after it returns TRUE for o once after running the randomizing function. Let us call this variable L, and we propose a value of 1095 days (3 years) for it.

In other words, when one ad tech calls a forDebuggingOnly method:

• with probability 999/1000 the API will not send a report, and furthermore that browser will ignore forDebuggingOnly from the caller ad tech for the next one year.
• with probability 1/1000 the API will send a report, at which point the browser will ignore forDebuggingOnly calls from any ad tech, for the next 3 years.

Based on these variables, and based on 2 reasonable assumptions we can make:

• The number of Chrome clients with the PA API enabled are around 2,000,000,000
• There being around 100 adtech companies integrated with our PA APIs in steady-state

We calculate that in legitimate scenarios like the ones detailed in the opening paragraphs of this reply that each participating adtech should be getting between ~4.7K and ~5.4K daily reports, if they choose to implement forDebuggingOnly at every generateBid and scoreAd instance. The variability is correlated to the number of participating adtech, the more participating adtech there are, the higher the likelihood that a given user is already within its lockdown period L Additionally, adtech can determine that forDebuggingOnly is only (or most) useful when certain criteria that are detectable by either generateBid or scoreAd are met. In these cases we suggest adtech to only call the forDebuggingOnly API in cases when it detects a critical situation that’s important to investigate so as to minimize the chance they are incurring a self-imposed cooldown period when calling forDebuggingOnly.

We also want to highlight the protections that we see against malicious scenarios with this approach. A malicious actor that knows that the sample rate is 1/f can choose to implement forDebuggingOnly by implementing a for…loop where i=f, with the intent of getting a report every single time they run scoreAd or generateBid. When the actor gets a report, their plan is to attach it to a unique ID they have created for this user/browser, and waits for the next, second instance of getting a report from the same user/browser to add to their profile. The cooldown parameter c protects the value of o, if it returns FALSE, from being re-evaluated for another c days, rendering the malicious intent of the for…loop meaningless. Now, if o returns a value of TRUE, then the actor has to wait 3 years before being able to get a report from the exact same user/browser. And if we layer on top the very small probability that after year 3 has elapsed the actor will get a report from the exact same user, we can see how these protections make these profile building efforts essentially unviable and untenable.

We believe that with this proposal, we can accomplish the goals we set out in our opening paragraphs. Any and all feedback is very appreciated!

@jonasz here you go

[JacobGo]

We are thrilled to see longterm support for these debugging APIs and look forward to the improved observability as we mature our integrations. I wanted to raise two concerns with the details of the above proposal.

In practice, we have observed some overhead when enabling debug codepaths due to the additional code profiling and report building. Given the highly latency sensitive worklet execution environment, we would recommend a mechanism to detect availability of forDebuggingOnly within the worklet to avoid wasting computation on building events that will never be sent.

Additionally, we are concerned about the shared lockout period given the threshold for critical situations may differ across adtechs. If one buyer decides to frequently invoke the API or unintentionally introduces a major bug which accelerates their call rate to 100%, should this lockout another buyer who needs to debug their own rare exceptions or sudden incidents?

[michaelkleber]

In practice, we have observed some overhead when enabling debug codepaths due to the additional code profiling and report building. Given the highly latency sensitive worklet execution environment, we would recommend a mechanism to detect availability of forDebuggingOnly within the worklet to avoid wasting computation on building events that will never be sent.

Hmm. There are two different things you might be asking here:

1. "Give me a mechanism that tells me whether a debug report would actually get sent."
2. "Give me a mechanism that tells me whether this browser would even consider sending a report right now, with the understanding that even if it would consider it, there is only a 1/1000 chance of sending it."

I think 1 would need to be an API that actually performed the die roll and triggered the cooling-off period 999/1000 of the time that 2 returned true

I'm not sold on either one of these, but which are you asking for?

Additionally, we are concerned about the shared lockout period given the threshold for critical situations may differ across adtechs. If one buyer decides to frequently invoke the API or unintentionally introduces a major bug which accelerates their call rate to 100%, should this lockout another buyer who needs to debug their own rare exceptions or sudden incidents?

If one buyer spams the API for whatever reason, the worst they could do is lock out 1/1000 of people for everyone else. The cooling-off period that happen 999/1000 times isn't shared state — it is only the 3-year lock-out that would let one ad tech affect another ad tech.

[JacobGo]

Thank you for the responses. Could you kindly elaborate on why triggering the cooling-off period is necessary when detecting API availability? Is the concern that we would have access to the 1/1000 device-sticky decision to truly send the debug report, and that this may influence or leak out of the internal worklet execution? I believe we're asking for (1), but without tripping the cooldown period given this (a) effectively incurs the statistical cost of always invoking the API and (b) may be a surprising side effect for all developers. I'm afraid this may incentivize us to always invoke the API if it's detected rather than save it for error states; alternatively we should just accept the overhead and restrict building the event messages to truly exceptional scenarios.

Great point about the difference between global lock-out and per-adtech cooling-off periods, I agree that the interplay of these successfully mitigates the impact of a spammy adtech.

One final note: as a user of forDebuggingOnly, I find the terminology slightly inverted here. I would expect "cooldown" to reflect the no-op state after a successful invocation of the API, whereas the lockout period may reflect the API being completely inaccessible in the first place.

[michaelkleber]

Thank you for the responses. Could you kindly elaborate on why triggering the cooling-off period is necessary when detecting API availability?

An API that is of the form "If I asked for a report right now, then would you send it?" would completely eliminate the 1-year cooling-off period, right? — after all, nobody would ever call the debugging API if they knew that it would not send a report. Your request would allow circumvention of all the "protections against malicious scenarios" that Alonso described above.

Or maybe I'm still misunderstanding what you're asking for?

On the other hand, I don't see any harm in an API of the form "Am I currently cooling down and/or locked out?" That would let you build your debugging requests much less often than without it, even though you would still only have a 1/1000 chance of sending each one that you built. @JensenPaul WDYT?

(Regarding "lockout" vs "cooldown", I personally feel like "lockout" feels more global, like "the door is locked", while "cooldown" seems more caller-specific, as in "you are over-heated, go take a walk and cool down and then you can come back and join the rest of us." But if other people have opinions on these or other more intuitive names for the two states, please share!)

[JacobGo]

Ah, I was assuming that the FALSE die roll was cast once per worklet function execution and there was no way to coordinate a loop based attack external to these functions. Thinking outside of that box, it does become clear why the check itself requires a cooldown. Any mechanisms to minimize the overhead of the API usage would still be welcome.

Overall, the statefulness of this API makes it more difficult to conceptually model an observability framework compared to traditional random sampling. I wonder if there might be issues here with a population more prone to exceptional circumstances gradually dwindling overtime due to the cooldown, as well as the true rate of an exception becoming invisible without a fully transparent sampling rate? I also worry about the longterm repercussions of an initial, overly lax threshold of exceptional events, e.g. an adtech accidentally locking themselves out of the API for a year.

[michaelkleber]

Thanks. I think the "Am I currently cooling down and/or locked out?" API would indeed help with minimizing the overhead, we'll explore that.

I wonder if there might be issues here with a population more prone to exceptional circumstances gradually dwindling overtime due to the cooldown, as well as the true rate of an exception becoming invisible without a fully transparent sampling rate?

I agree with this concern, but I haven't come up with any other way to preserve the privacy goals.

I also worry about the longterm repercussions of an initial, overly lax threshold of exceptional events, e.g. an adtech accidentally locking themselves out of the API for a year.

Yes, great point, that does seem like it's too easy to accidentally shoot yourself in the foot.

Instead of a 1-year cool-down when you don't get a report, I wonder if we could instead have a shorter timeout, like 1 week, that would trigger 90% of the time, and a 1-year timeout the other 10%. Then even if an ad tech shipped a bug that asked everyone to send a debug report all at once, they would recover the ability to debug on 90% of their traffic a week later. (All percentages and time durations subject to change, but at least it's an idea.)

[michaelkleber]

Okay, I've done a little simulating of this idea of the anti-footgun two-cooldowns idea — thank you Google Sheets for the "Iterative calculation" capability in the File>Settings>Calculation menu.

Suppose that when you ask for a debug report in a Chrome instance which is not in the cool-down or lock-out state,

• A debug report gets sent 1/1000 of the time, and that browser enters global lock-out for 3 years
• Otherwise, no report gets sent, and:
  • there is a 90% chance the calling ad-tech enters a 2-week cool-down
  • there is a 10% chance that the calling ad-tech enters a 1-year cool-down

Which is to say: if you accidentally push into production a bug that asks everyone in the world to send you a debug report, you would regain your ability to do selective debugging on 90% of browsers after two weeks, instead of after one year.

In that case, with 100 ad techs spamming the API as much as possible, each one gets around 6500 debug reports per day per billion Chrome instances.

If there were only a single ad tech using the API, they would instead get around 20K reports per day per billion, so the global lock-out mechanism cuts the number of reports to about 1/3 of what it would be otherwise. The truth will probably be somewhere between those two extremes.