Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Master Feature] Support IAB Consent Framework using amp-consent #15653

Open
jasti opened this Issue May 29, 2018 · 40 comments

Comments

@jasti
Copy link
Collaborator

jasti commented May 29, 2018

At the moment, consent within amp-consent is binary. Either the user can accept consent for all vendors on the page or reject consent for all vendors on the page.
This feature provides the ability for a user to accept or reject consent on a per vendor basis. Further, this could also be enhanced to support consent on a per vendor and per purpose basis.
for e.g. a user should be able to give consent at a granular level to Vendor A for the purpose of analytics but reject consent to Vendor A for the purpose of ads.
Master issue is #15651

AMP will provide the generic framework for any 3rd party CMP to directly integrate with amp-consent to manage consent workflows.

If you are a CMP, please get in touch with us on this issue.

@jasti jasti added this to the New FRs milestone May 29, 2018

@jasti jasti added this to To do in Consent Tools via automation May 29, 2018

@jasti jasti changed the title Support per vendor consent using amp-consent [Master Feature] Support IAB Consent Framework using amp-consent Jul 3, 2018

@jasti jasti added this to Next Up in AMP HTML Project Roadmap via automation Jul 16, 2018

@jeffnordlund-ghost

This comment has been minimized.

Copy link

jeffnordlund-ghost commented Jul 27, 2018

Hey, I work for a CMP and would love to know how you are planning to implement the IAB support and am happy to provide any feedback if needed.

@jasti

This comment has been minimized.

Copy link
Collaborator Author

jasti commented Jul 30, 2018

@jeffnordlund-ghost thanks! We've made some progress and @zhouyx is working on documenting the design for community feedback. @zhouyx please post design here when ready.

@fchristant

This comment has been minimized.

Copy link

fchristant commented Jul 31, 2018

I think this feature is much needed in order to meet GDPR requirements which explicitly states that a user must be able to give individual consent on a per processing purpose basis.

I don't really understand how the idea is vendor driven, imho it is purpose driven. A user gives consent to a data processing purpose, not to a specific vendor doing the processing? I suppose it is possible to have two vendors on the same page executing the same processing purpose, whereby a user can give consent to each vendor, but that seems a bit of an edge case to me.

@felixarntz

This comment has been minimized.

Copy link
Contributor

felixarntz commented Jul 31, 2018

Hey there! I have been working on a cookie consent plugin for WordPress that supports AMP via amp-consent, and I'd like for it to support more granular cookie control as well (it already does for non-AMP sites). I agree with @fchristant that I'd expect those controls to work per cookie category / purpose, not vendor. Per-vendor management might be useful as way, in that case I think both should be covered.

As an example, here is what cookie types my cookie consent allows to be specifically allowed/disallowed:

  • Functional (automatically-checked, can't be disabled when accepting cookies)
  • Preferences
  • Analytics
  • Marketing
@jasti

This comment has been minimized.

Copy link
Collaborator Author

jasti commented Jul 31, 2018

@fchristant thanks for the feedback. We were worried initially about being able to have the capacity to store vendor and purpose information in local storage but further analysis shows we can. So we'll simply rely on the CMP passing AMP with the "daisy bit" as defined by the IAB: https://github.com/InteractiveAdvertisingBureau/GDPR-Transparency-and-Consent-Framework/blob/master/v1.1%20Implementation%20Guidelines.md#consent-management-provider-guidelines- which supports both vendor and purpose.

@jeffnordlund-ghost

This comment has been minimized.

Copy link

jeffnordlund-ghost commented Jul 31, 2018

Yeah, just leaving that generic would be best and just set up a communication mechanism for that information. Currently the IAB supports consent by purpose, or by vendor, but the vendor is all or nothing (no granular purposes on a vendor). But, rumor has it they are looking into the granular purposes by vendor support (not really sure why they didn't go that way to start).

For us we are supporting consent via the iframe component because our UI is so rich and we would need to basically re-write it to natively support amp-consent. So please keep that in mind and enable this through the message flow for the iframe.

Since the native IAB __cmp interface is able to support cross-frame messaging that should be relatively straight-forward to accomplish.

@jasti

This comment has been minimized.

Copy link
Collaborator Author

jasti commented Jul 31, 2018

@jeffnordlund-ghost We weren't planning on supporting the IAB version from an external iframe because the external iframe will then need to be page render blocking which is overall a pretty bad experience of the user from a latency standpoint.

We'd recommend considering adding native support unless you can think of alternatives that ensure good performance. @zhouyx anything else to add?

@jeffnordlund-ghost

This comment has been minimized.

Copy link

jeffnordlund-ghost commented Jul 31, 2018

I don't understand why that would be any more blocking then normal consent. I assume you are going to store the consent string in local storage as you do the consent marker now, correct? Also, the __cmp interface is designed to be non-blocking to page render. It will prevent ads from firing until a response is returned, but the page can render as normal. It just queues up the calls made to it until they can be answered. It shouldn't be that difficult to just set up a message interaction to route those calls to a calling iframe.

The problem we have with the standard amp-consent is it's just too simplistic. We deal with huge global brands where they support dozens of languages and most countries. Our platform simplifies that for them, centralizing everything from messaging to keeping their vendor lists up-to-date. It just wouldn't be possible to do that in the typical amp-consent tag, there is too much dynamic stuff happening.

@jasti

This comment has been minimized.

Copy link
Collaborator Author

jasti commented Jul 31, 2018

I agree that it's not different from the current amp-consent solution but the external iframe solution was meant as a stop-gap solution for publishers who have simplistic consent gathering UIs.
For CMPs, we recommend native support. Note that you will have full control over the UI in this new version. I'd suggest we continue discussing this topic once @zhouyx publishes a design for how CMPs can integrate?

@jeffnordlund-ghost

This comment has been minimized.

Copy link

jeffnordlund-ghost commented Jul 31, 2018

Sure. I was not aware you had plans to make the consent UI support more robust. I would very much like to see what is on the roadmap for that so we can plan accordingly.

@zhouyx

This comment has been minimized.

Copy link
Collaborator

zhouyx commented Jul 31, 2018

Yes we are planning to support the standard IAB purpose list and vendor list. But currently has no plan to support the granular purposes by vendor because it requires much space to store the consent and from the feedback we gather from some CMP they always collect purposes for all set of vendors. Please let us know if there's requirement for more granular purposes per vendor.

@jeffnordlund-ghost As @jasti pointed out, we don't have a good way to ensure good performance via iframe. Which is very important especially given the UI dialog is blocking user from consuming the page. Could you give us an example of your UI? And we can see what's the best way to support your use case. Thank you!

@jeffnordlund-ghost

This comment has been minimized.

Copy link

jeffnordlund-ghost commented Jul 31, 2018

You can go to a number of places to see our consent ui. https://www.cosmopolitan.com/uk/, www.vmware.com, www.crownpeak.com, cbs.com, most nestle sites. Some of them require you come in from an EU country to see the UI though.

Our UI isn't dramatically different from other CMP's. But it is fully customizable (colors, fonts, size, which buttons are shown, what those buttons look like, etc.) and it automatically picks up on things like language settings and current geo so our customers can deploy a common script on every one of their properties and then manage everything from our interface. We also have a process to automatically scan their pages and detect vendors so they don't have to keep updating their vendor lists. So the more of that dynamic nature that can be supported the easier it is for our customers to deploy and manage. But most of that requires scripts to function.

Additionally, we need to be able to do reporting. Currently we report by dropping image pings with the information we need for reporting. But reporting is crucial for compliance with regulations like the GDPR. Our current scripts will log a number of actions, including consent, with enough information to satisfy the EU regulators if necessary.

@zhouyx

This comment has been minimized.

Copy link
Collaborator

zhouyx commented Jul 31, 2018

Thank you @jeffnordlund-ghost This is really helpful.

We also have a process to automatically scan their pages and detect vendors so they don't have to keep updating their vendor lists. So the more of that dynamic nature that can be supported the easier it is for our customers to deploy and manage. But most of that requires scripts to function.

This is an interesting requirement that we haven't heard from other CMP before. I wonder how do you scan the page to detect vendors, especially given that some vendors are not directly included in the page but requested by another vendor. What kind of information do you expect from AMP to help you achieve that? Because even with the iframe approach you won't be able to run script on the parent page.

Our current scripts will log a number of actions, including consent, with enough information to satisfy the EU regulators if necessary.

AMP do supports some sort of reporting. For example it send a POST request to the configured endpoint with user's decision. Is that sufficient?

@jeffnordlund-ghost

This comment has been minimized.

Copy link

jeffnordlund-ghost commented Aug 1, 2018

In regards to the page scanning there isn't anything the AMP framework needs to do to support that. But what that means is we get a frequently changing list of vendors and that is what we need to support. So some way to pull in the list of vendors/tags (both IAB and non-IAB) would be really helpful. If our customers need to re-add that to their AMP pages as a static list that can become a maintenance nightmare. The IAB is planning to support something that might make this easier, a publisher.json file that allows the list of vendors to be stored in a json file. Support for something like that might be enough to enable this.

It would be nice if you supported some additional events. We frequently get requests from customers for things like opt-in rates. So we need to store both the number of impressions as well as the number of users who accepted or declined. They ask us for details on which vendors are being opt-ed out of, or how often the user elects to opt-out of all vendors (for some scenarios). Our expectation is they will ask for similar reporting for the IAB platform: which purposes are enabled/disabled, how often does a user opt out of everything, how often do they even look at the choices, etc. If you could make the reporting more generic instead of just listing specific actions that would give people the flexibility to report on what is important to their customers.

Also, it would be really nice if you were flexible enough to support GET requests as well as POST's. We have our reporting set up the same way as many advertisers do where we just drop image pings, which are all GET requests obviously.

Other nice to have: Support for a settings file. This would allow our customers to deploy a central amp script and let the settings dictate things such as which geo's need consent, which vendors/tech are on the page, what does the consent dialog look like. Things like that.

@zhouyx

This comment has been minimized.

Copy link
Collaborator

zhouyx commented Aug 1, 2018

@jeffnordlund-ghost Thank you for the information. Again this is really helpful!

To support different analytics events, and different request method, we are planning to support a CMP based template. Then you can use or component to collect analytics with much flexibility.

Other nice to have: Support for a settings file. This would allow our customers to deploy a central amp script and let the settings dictate things such as which geo's need consent, which vendors/tech are on the page, what does the consent dialog look like. Things like that.

Yes we have onConsentHref that's supposed to support features like this.

@jeffnordlund-ghost

This comment has been minimized.

Copy link

jeffnordlund-ghost commented Aug 1, 2018

I assume onConsentHref is something new since I don't see any docs for it yet.

If you can start to build in support for GET requests that would be great. We don't use servers for 99% of our stuff, just using our CDN to distribute the scripts and settings. This makes our scripts faster and our uptime is basically 100% as opposed to using a server. So supporting that would be great for us.

@janwinkler

This comment has been minimized.

Copy link
Contributor

janwinkler commented Aug 13, 2018

We are a CMP too, how can we contribute?

@consentmanager

This comment has been minimized.

Copy link

consentmanager commented Aug 13, 2018

:-)

@jawadst

This comment has been minimized.

Copy link

jawadst commented Aug 14, 2018

@jasti To double down on Jeff's comments and list a few more requirements that we (as a CMP) have for our regular web/in-app SDKs and that are actively used by publishers today:

  1. Ability to control the workflow (when to collect consent, what the different banners / popups look like and how the user goes from one to the other, different languages, etc.). As you mentioned, being able to provide a ready-to-go template per CMP would go a long way vs the existing situation where every publisher needs to rebuild the consent UI.
  2. Use our own list of vendors and purposes (we rely heavily on the list provided by the IAB but also need to support custom vendors and purposes that are not part of the IAB). This might also require being able to store more information than just the IAB consent string.
  3. Be able to condition the loading of tags at the vendor level for a given set of purposes so this can be driven by the consent string and does not necessarily need to be a CMP-specific approach. It can rely on the existing data-block-on-consent attribute, assuming it gets the adequate level of granularity.
  4. Reporting of all events to our back-end (consent status, clicks on various elements of the UI, etc.)
  5. Support for different ways of giving consent. Depending on the country and the DPA, publishers might want to collect consent on scroll, navigation or click on the banner.

Also happy to help review any documentation/design that would be targeting CMPs.

@zhouyx

This comment has been minimized.

Copy link
Collaborator

zhouyx commented Aug 14, 2018

@jawadst Thank you for the information. At this point we are leaning to let CMP implement the logic within an iframe, and then AMP runtime can help to store and pass the consent string to components on the page.

Since you mentioned that you will be supporting vendors that's not necessary of of the IAB list. Could you please provide more information on what exactly are you storing, and passing to vendors. My concerns are

  1. AMP has very strict rules on the size of storage that can be used for this purpose. I want to make sure the consent string you are going to store is space efficient.
  2. Your consent string can be passed to every vendors that integrates with AMP. I want to make sure that all vendors can understand the response from CMP.

Thanks

@jawadst

This comment has been minimized.

Copy link

jawadst commented Aug 14, 2018

@zhouyx Thanks for your answer!

The iframe option would work great. We are currently using the existing <amp-consent> with an iframe and that works pretty well.

For non-IAB vendors, we usually store the consent information for each of them with a structure that looks like this:

{
  enabled: ['custom_vendor_1', 'custom_vendor_2'],
  disabled: ['custom_vendor_3']
}

While that structure is less efficient than the IAB one, it is usually only used for a small number of vendors (I think our publishers use ~10 custom vendors on average).
The IAB specification allows using a format similar to the consent string for publisher-specific vendors (https://github.com/InteractiveAdvertisingBureau/GDPR-Transparency-and-Consent-Framework/blob/master/Consent%20string%20and%20vendor%20list%20formats%20v1.1%20Final.md#publisher-consent-stored) so that might be used as well. To be fully transparent, we like to have string IDs for those vendors but that might be very specific to our CMP and allowing a second consent-string that would be CMP/publisher-specific and can be referenced for blocking/unblocking tags might be enough.

Those custom vendors are important because the IAB mostly covers advertising vendors but not all others (social, analytics and A/B testing are the main ones we are seeing). It's especially important to have a way to cover at least all the different categories that have custom AMP components (so not just advertising but also analytics and social).

As to what we pass to these vendors, there are two cases:

  1. If they support receiving consent information through their own API, we provide that (that's the case for DFP or AdX, for instance: https://support.google.com/admanager/answer/7678538). That's similar to what the current AMP consent does I think so there might not be anything special to do except being able to keep the consent at the vendor level.
  2. If they don't support receiving consent information at all, we simply block the tag. That's the current behavior of AMP consent if we could somehow have a condition on consent specifically for a custom vendor with the data-block-on-consent

Hope this helps, happy to provide more details or examples if needed. Our documentation is public if that can help better understand the use case: https://developers.didomi.io/privacy-management/consent-management/consent-notice/vendors-and-purposes

@jawadst

This comment has been minimized.

Copy link

jawadst commented Aug 14, 2018

@zhouyx
Another point that is usually pretty important to publishers is being able to load non-personalized ads without consent and then -sometimes- reload ads after consent has been given (it usually has an impact on viewability so not every publisher makes that choice). That's usually a configuration that is specific to advertising tags.

DFP, for instance, has support for non-personalized ads (although consent for cookies is still required) and I think that a lot of vendors are going to start offering more options for non-personalized ads eventually.

I think that the current AMP consent somewhat supports that with the combination of a fallback action with "dismiss" status but:

  1. I am not sure what the interpretation of "dismiss" is for vendors and if an extra status "unknown" would not be useful. Maybe "dismiss" is enough though but that could be made more explicit in the documentation.
  2. I don't think there is an option to reload some tags after consent is given.
    The scenario "load ads without consent then reload after consent" would be a combination of fallback=0 and "reload after consent".
@zhouyx

This comment has been minimized.

Copy link
Collaborator

zhouyx commented Aug 15, 2018

Thank you @jawadst for all the details.

For non-IAB vendors, we usually store the consent information for each of them with a structure that looks like this: {
enabled: ['custom_vendor_1', 'custom_vendor_2'],
disabled: ['custom_vendor_3']
}

Due to the fact that AMP page can be cached and not served from the origin domain. There's very strict usage on the storage space.
I understand that your customized vendors number is small, but still I am worrying about the storage usage. Could you provide me an estimate on the maximum usage of storage in bytes? Currently we are thinking about a maximum of 300 bytes storage upper limit. Not sure if that will work for you.

I am not sure what the interpretation of "dismiss" is for vendors and if an extra status "unknown" would not be useful. Maybe "dismiss" is enough though but that could be made more explicit in the documentation.

It's up to the vendor that how they interpret 'dismiss'. We have vendors that decide to not serve ad on dismiss, but serve npa on 'reject'. The different to AMP is that, with 'dismiss' AMP will never store the decision.

I don't think there is an option to reload some tags after consent is given.
The scenario "load ads without consent then reload after consent" would be a combination of fallback=0 and "reload after consent".

You're correct! There's no such reload option for AMP now. Publishers can choose to serve a npa while asking for consent. And when someone visit the site next time a personalized ad will be served. But that's depends on the publishers' configuration.

@zhouyx

This comment has been minimized.

Copy link
Collaborator

zhouyx commented Aug 15, 2018

I wonder if it is even possible for CMP to store some related information on server end?

@consentmanager

This comment has been minimized.

Copy link

consentmanager commented Aug 16, 2018

@zhouyx The IAB consent string is usually less than 60 or 70 bytes. Anyhow, with the next versions of the consent string there are some additions that can extend the consent string up to 700-800 bytes.

@zhouyx

This comment has been minimized.

Copy link
Collaborator

zhouyx commented Aug 16, 2018

@consentmanager Thank you for the heads-up. I didn't know the next version of the consent string expands its size so much : ( Is there a doc that I can refer to? Thank you!

@jasti jasti moved this from Next Up to In Development in AMP HTML Project Roadmap Aug 16, 2018

@consentmanager

This comment has been minimized.

Copy link

consentmanager commented Aug 16, 2018

@zhouyx storing info on server end: probably possible (e.g. give the user a cookie with an id and save the consent string on servers) but not yet supported by the iab framework

@jawadst

This comment has been minimized.

Copy link

jawadst commented Aug 28, 2018

@zhouyx Storing consent on the server-side is possible independently from the IAB framework but most CMPs would rather not have to store anything server-side for availability and speed questions. Especially in AMP, it is extremely important to guarantee that everything is stored directly on the device to avoid extra HTTP requests and increased latency for the user (especially when the CMP views are likely to be blocking for the user).

@jeffnordlund-ghost

This comment has been minimized.

Copy link

jeffnordlund-ghost commented Aug 28, 2018

the biggest problem with storing info server-side as a CMP is identifying the user the information belongs to. without some kind of specific user identifier it isn't possible to map a server request to the consent information and your standard browser request doesn't contain enough information to accurately identify a specific user from random sites on the internet. that, and performance, are the main reasons we use either cookies or local storage for this information.

@zhouyx

This comment has been minimized.

Copy link
Collaborator

zhouyx commented Aug 29, 2018

Especially in AMP, it is extremely important to guarantee that everything is stored directly on the device to avoid extra HTTP requests and increased latency for the user (especially when the CMP views are likely to be blocking for the user).

@jawadst This is a great point. I know some would still want to make a request for additional information even with stored value. Is that true for your case? Can I assume that once there's previous stored consent string, AMP runtime can simply use that without sending request to CMP?

without some kind of specific user identifier it isn't possible to map a server request to the consent information

@jeffnordlund-ghost You're correct it's impossible to identify user on server side right now. But it can be supported if we feel server side storage is the way to go. What concerns me more is that do you feel it's OK for you to store an user's identifier as a CMP in general?

@mobiluck

This comment has been minimized.

Copy link

mobiluck commented Aug 29, 2018

@zhouyx the amp runtime should/must load the cmp code every time on every page, even if the value is already stored. the code will be providing the functionality/API for vendors, hence its necessary to load it even in cases where consent is already given

@jeffnordlund-ghost

This comment has been minimized.

Copy link

jeffnordlund-ghost commented Aug 29, 2018

No, that's kind of the point of not doing things server side. If we do that then we are forced to store some kind of user identifier and we have no desire to do that. All we want to do is provide users a mechanism to exercise their choices and then communicate those choices to the publisher.

@hhhjort

This comment has been minimized.

Copy link

hhhjort commented Sep 26, 2018

Good to see AMP moving in the direction of the IAB consent framework. One use case I haven't seen explicitly mentioned is an RTC call to a vendor that will then proxy additional calls to additional vendors. (Example: prebid) That will require passing the IAB consent string (or the same information in some other format) in the RTC call so the vendor can properly support the GDPR requirements in whatever it does with the request.

@jeffnordlund-ghost

This comment has been minimized.

Copy link

jeffnordlund-ghost commented Sep 26, 2018

I was under the impression that forwarding the consent string through to downstream calls is the responsibility of the vendor. So an auction vendor would be responsible for sending the string to the auction winner. I don't think there is anything special that would be required in AMP to support that.

@hhhjort

This comment has been minimized.

Copy link

hhhjort commented Sep 26, 2018

Well, the vendor needs the consent string to be able to forward it. If the vendor is being called via AMP RTC, then AMP needs to be able to provide the consent string in the RTC call for the vendor to have it.

@jasti

This comment has been minimized.

Copy link
Collaborator Author

jasti commented Oct 15, 2018

@keithwrightbos are there plans for DoubleClick to pass the consent string to RTC vendors?

@lannka lannka removed their assignment Oct 22, 2018

@jdelhommeau

This comment has been minimized.

Copy link

jdelhommeau commented Feb 20, 2019

Hi @jasti ,

Is there any update on supporting consent string passed in RTC config? Seems to be a blocker for prebid support in AMP.

@zhouyx

This comment has been minimized.

Copy link
Collaborator

zhouyx commented Feb 22, 2019

Hello @jdelhommeau I talked to @jasti today on this topic. We can definitely add support to pass consent string along with the RTC request.

Trying to understand your requirement better? Are you working with the prebid server, that can consume the consent string along with the RTC request.

@hhhjort

This comment has been minimized.

Copy link

hhhjort commented Feb 25, 2019

I imagine the best way to support this is to have a macro that will be populated with the IAB consent string if it is available. Ideally also a GDPR flag to signify if the viewer is in a location where the GDPR applies. If these two can be placed in the query string, it will not be difficult to modify prebid server to read and make use of them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.