Incorporate market impact and user feedback as a part of FLEDGE experiment goals

[lbdvt]

The FLEDGE proposal states:

Chrome expects to build and ship this first experiment during 2021. The goal is for us to gain implementer experience, and for the ads ecosystem to evaluate its usability, as soon as it is feasible to do so.

While gaining implementer experience is a valuable thing, we think that any experimentation should also incorporate the two goals below, as they will be required for any third-party cookies replacement:

• Assess the economic impact, e.g. by measuring the impact on CPM
• Assess the user satisfaction, e.g. is there a higher user satisfaction with ads provided through FLEDGE compared to ads provided with the help of 3p cookies?

Would you consider adding these two objectives as "experiment goals" for FLEDGE?

[michaelkleber]

Hi Lionel! I would have considered both of your goals as pieces of letting the "ads ecosystem [...] evaluate its usability". In general, browsers implement APIs and the web developers of the world try them out and give us feedback on how well they work, and "how well they work" is quite broad, covering anything the users of the APIs are hoping to get out of them!

[lbdvt]

Hi Michael,

Following the last IWABG conf call, I understand Chrome's position of not willing to define metrics for all participants, but providing the tools for everyone to make their own experiments.

At Criteo, we therefore propose to IWABG participants -advertisers, publishers, ad tech providers, privacy experts, individuals, scholars- to discuss, in this thread, what they would consider as good FLEDGE experiment metrics. Our goal here is not to come up with a unique set of metrics for everyone, but rather to trade ideas and see what comes out.

I'd like to point out the distinction between metrics, something measurable as average CPM for instance, and use cases, such as "lookalike targeting" or "niche advertising". I suggest that we focus on metrics here, as use cases will differ from one actor to another (even though different use cases may call for different metrics).

So starting the conversation with Criteo, we think of using two metrics on a variety of use cases:

• The spend on ads, as it captures 'the utility' value of the ads. It's also a good proxy for publishers' revenues.
• A user Net Promoter Score, NPS, to measure user satisfaction.

To compute an NPS, one runs a multi-choice survey with a numeric value assigned to each choice, usually some choices with a negative value, and some with a positive one. The final metric is the sum of all answers' values.
We envision running that kind of survey on a small percentage of ads displayed using FLEDGE, and on a set of ads displayed using 3p cookies as a control group. After a short display time, the survey would cover the ad and feature questions like "What are your feelings about this ad? Negative / Neutral / Positive". The Net Promoter Score is then calculated by counting the "Positive" answers minus the "Negative" ones. Negative answers could be followed by an additional question, for further investigation. Using this tool, we hope to be able to measure the impact of features such as "micro-targeting" prevention.

Dear participants to the IWABG, what metrics are you planning to look at when experimenting with FLEDGE? Why?

[lbdvt]

The CMA lats December publication offers a new framework for that discussion, so closing the issue.