Agenda Request: Experiment to better understand the effect of DP on advertising decision making

[eriktaubeneck]

Agenda+: Experiment to better understand the effect of DP on advertising decision making

We’ve been exploring how to measure the effect of Differential Privacy at different parameterizations, and are seeking feedback on a test that we’ve designed. We are attempting to isolate the effect of adding noise on decision making. More specifically, given some decision making process D(x) (for some input variables x), how does D(x’) change for noised x’?

We build this small game which simulates conversion measurement results from hypothetical campaigns. Those results are then duplicated and have Laplace noise added to make them differentially private. These results are all presented to the user, in random order, with a simple binary decision: increase spend or decrease spend. After the user completes the round, it will measure how often the same decision was made for both x (the true simulated data) and x’ (the noised version of x).

We expect that as ε increases, the ability to make the same decision should approach 50% (pure random chance.)

Time

45-60 min
For this agenda item, we want to cover three areas:

1. An overview of the game and the test we want to run with it. (15-20 min)
2. Feedback on the design of the game (e.g. parameterization, simulation process.) (15-20 min)
3. Feedback on running an experiment (e.g. who to recruit, how many participants we’d want, help finding participants.) (15-20 min)

Links

Differential Privacy Game (early prototype, open issues here if you find bugs!)

[alexWhitworth]

We'd be quite interested in this at Pinterest

[eriktaubeneck]

Slides from the presentation

[jgoodknight]

I played around with this and it's really cool! I had a couple thoughts.

I wonder if there would be a way to build in some kind of constraint? Perhaps for every round you have 10 tokens to spend to "exploit" and you have to chose which campaign they go to, and the system spends another 10 tokens uniformly to "explore"? I realized that in the absence of some kind of budget constraint, the 'correct' answer is just to spend money on any campaign that gets you conversions. Or rather, any campaign that gets you conversions at a cost-per-acquisition lower than your profitability target.

I am guessing, however, that CPA is probably too difficult to simulate though and then how do you even think about simulating a user's profitability target. Thus making conversion rate a decent proxy.

Also I thought it would be easier to make decisions if the numbers at the decision screen were presented as a rate/ratio and started playing wasn't clear to me if I was supposed to assume every campaign had the same number of impressions or not although then. Presenting it as a conversion rate would also allow you to take into account the spend choices adjusting the number of impressions from the previous experiment round if you want to.

I could probably get some privacy engineers at snapchat interested in playing the game for a while too so let me know! Who it is easy to recruit (privacy and software engineers) might be different than who you want to recruit in a perfect world (marketers?)