Support for CCPA opt-outs

[darobin]

Describe the new feature or change to an existing feature you'd like to see

On January 1st, 2020 the CCPA comes into effect. One of the provisions of the law is that residents of California must be granted the ability to opt out of their data being sold. It is not immediately clear how to implement such functionality in AMP. Having looked at various approaches, at least the following items are potential issues or blockers:

• amp-geo only works at a country level; in order to offer California-specific affordances (for those who choose to implement it that way) it would require the ability to also provide US state-level results (for instance by placing a separate state-specific class).
• There can only be one amp-consent instance on page, so anyone already using amp-consent for other purposes, typically GDPR, cannot also use it for CCPA. (CCPA is an opt out mechanism, but this could be represented through a consent affordance triggered by a button, on which the options are reject and close). amp-consent was designed to support multiple instances but that seems to have never been implemented.
• Both amp-analytics and amp-ads seem to support opt-in behaviour whereby they are off until an amp-consent is agreed to, but they do not seem to support opt-out behaviour in which they run until an opt out is effected (equivalent to an amp-consent reject) at which point they can be selectively disabled or modified. They would need to support that behaviour.
• amp-ads does not currently appear to have the ability to pass the IAB flag to signal opted-out users.

Given the short time frame remaining, we would appreciate reassurance that the AMP team intends for it to be possible to run AMP legally in 2020.

Describe alternatives you've considered

We have tried to hack this together every which way and it does not seem possible to make it work.

[dvoytenko]

/cc @ampproject/wg-analytics

[zhouyx]

@jeffjose is looking into this from the requirement point of view.

[jeffjose]

Hi @darobin, Thanks for starting this thread, and your detailed analysis on what's possible with AMP today.

You're right -- we can have better controls to support CCPA in AMP. We're kicking off an initiative to support CCPA affordances in AMP. We intend to give publishers such as yourself the mechanisms needed to have AMP pages compliant in 2020.

I'll update this thread when I have more details to share.

[darobin]

Hi @jeffjose, thanks a lot for getting back to us! I hate to bug you but do you have a few more details beyond "in 2020"? The law comes into effect Jan 1st and we need time to adjust the code, test, deploy before that. Thanks!

[jeffjose]

Normally I'd shy away from giving deadlines, but optimistically we're trying to get a solution out by Nov 2019. Hopefully that gives you enough time to update your pages, test and deploy them.

[jeffjose]

@darobin I pinged you on slack (amphtml.slack.com) with a question. Would you mind popping over there and taking a quick look?

[tobie]

@jeffjose, @darobin: were you able to catch-up on Slack?

November is already upon us!

[jeffjose]

Yup! And we met few weeks back as well.

…

On Sun, Nov 3, 2019, 11:34 PM Tobie Langel ***@***.***> wrote: @jeffjose <https://github.com/jeffjose>, @darobin <https://github.com/darobin>: where you able to catch-up on Slack? November is already upon us! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24910?email_source=notifications&email_token=AABKH5BD7UNQNXSGNI7AWF3QR3OQPA5CNFSM4I5TMADKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC5T7FI#issuecomment-549142421>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABKH5F74BI7YAY4HCSGL4LQR3OQPANCNFSM4I5TMADA> .

[tobie]

That's great to hear. A summary of those conversations here would be useful to the community at large. :)

[jeffjose]

For sure. We're still finalizing the design and once we have that locked down, I'll share an update here. The gist of the meeting was talking thru how NYT was thinking about CCPA which wont be super-interesting to the wider community.

…

On Mon, Nov 4, 2019 at 5:13 AM Tobie Langel ***@***.***> wrote: That's great to hear. A summary of those conversations here would be useful to the community at large. :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24910?email_source=notifications&email_token=AABKH5DWNBZUWT6QWWVUXBTQR4WFZA5CNFSM4I5TMADKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC53U7Y#issuecomment-549173887>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABKH5FXMVHT4ZBL54PMHYDQR4WFZANCNFSM4I5TMADA> .

[darobin]

We have mostly rehashed what is in the issue and ruled out alternative options. We're looking forward to seeing how that translates to implementation.

For clarity though, this is not an NYT issue, this is an issue for essentially everyone running AMP.

@jeffjose You were asking about the actual enforcement date, we have received confirmation from the CA DoJ that it will indeed be Jan 1st.

[tobie]

We're still finalizing the design and once we have that locked down, I'll
share an update here.

Is there any reasons for this process not to happen in the open?

The gist of the meeting was talking thru how NYT was thinking about CCPA which wont be super-interesting to the wider community.

This is an issue that’s affecting pretty much everyone in the industry. Of course understanding how you're thinking about solving it is important to the wider community. It’s on the AC agenda today, for example.

As a rule of thumb, it’s safer to broadcast what’s going and allow the community to decide what’s of interest to them rather than to make assumptions about this.

Designing in the open is also a much better strategy to gather broader input and get more external contributors involved.

[zhouyx]

Good point on share the design ideas in the thread and get feedback. Thank you @tobie for the suggestion.

Two major challenges we are facing.

Challenge 1. opt-out
opt out is difficult because unlike opt-in decision, opt-out decision in theory should not be purged from storage. This is extremely difficult with AMP's limited localStorage space. AMP's localStorage is implemented by the AMP viewer, and it's up to the AMP viewer to purge unused storage entries without advance notice. We can improve the AMP viewer's implementation to support LRU logic and also expand the storage space for AMP, but this issue will still exist.
Client side storage is preferred because otherwise there will always be a roundtrip delay to render any AMP pages (even if users are not from EEA or CA). We want to get publisher's feedback on the AMP localStorage limitation, and the risk of opt-out decision not being persisted.

Challenge 2: Handle multiple consents.
In the case there are multiple user consents (GDPR and CCPA). It's should be up to publisher or CMP to decide what to respect. This may involve some complicate logic including interpreting consent string and return different result based on users geo location. AMP may provide a very conservative way to combine boolean based consent state. But for more advanced features, one idea is to have publishers use <amp-script> to combine multiple consent decisions.

Ideas, feedback and requests are very welcomed!

[tobie]

/cc @ampproject/ac: if you're representing constituencies affected by CCPA opt-out requirements, please chime in. Thank you.

[tobie]

Also adding @MelSumner, @TedShuter, @aghassemi, @mjwettergreen, @jervay, and @msteffan (who I just realized weren't added yet to the @amproject/ac team).

[AramZS]

Hey, chiming in from the perspective of The Washington Post:

We really need this ASAP. Thanks.

[tobie]

@AramZS do you have feedback on the two challenges @zhouyx listed in #24910 (comment)?

[darobin]

With respect to the two challenges that @zhouyx outlines:

1. Complying with the law is actually more important than performance. As we said on the call, we see the localStorage version as a local optimisation but not a good overall solution. We want to be able to save opt outs on our end (so that we can sync across products) and to load from our data as well. So saving to and pulling from us would be ideal. If the amp-consent is keyed of geography, it should be possible to only load when in the applicable geography.

2. This is actually pretty much the same decision. To the extent possible, the client side should not be making compliance decisions, that's just too many products to change. Combining multiple types of consent if and when multiple geographies apply is something that is great to centralise on the server and that AMP could simply load from.

[AramZS]

Responding to @tobie as requested:

Challenge 1. opt-out

I agree with what is said above, we need to be able to have consistency in the opt-outs across platforms. If a user has opted-out on washingtonpost dot com, arrives on a WashPost AMP via Search and that AMP has amp-access or amp-subscriptions user IDs that are logged into our systems then we need to be able to make a decision on the logged-in users data before any possible tracking or ads are loaded. If the user is not logged in, and they log in via AMP and opt-out in AMP we need to receive a data payload that allows us to register that opt-out across all our systems. As such we have the following as requirements:

• We must know if the user is logged in before anything that tracks them is loaded.
• We must know if a logged-in user is opted out in our site (outside of AMP Cache/Viewer) before anything that tracks them is loaded.
• We must know if a logged-in user has taken the action to opt-out from within AMP and be able to align that with their login inside our own systems.
• If a non-logged-in user opts out on AMP's localStorage that opt-out should be stored persistently within AMP localStorage from that point forward and there should be an event registering that opt-out with our site along with the AMP reader id.
• Finally I am pretty sure that if a non-logged-in user arrives on an AMP page it will need to send their AMP Reader ID to our API in order to assure we have not recorded a previous opt-out that would need to be retained.

I think it would be reasonable to no longer 'phone-home' for that opt-out's status once it is set and sent to our API.

If we want to find the most performant option, a solution might be: once the user's opt-out is registered and for as long as it is persistent in the AMP Viewer's storage the opt-out is the user's assumed state without an additional phone-home on each load.

However, we'd want to register changes in that status so an async call to a publisher-based endpoint on every AMP view after the load to check if logged-in users have retained their opt-out status since the last AMP visit would be required.

Challenge 2: Handle multiple consents.

I agree this is a decision best made by checking the AMP Reader ID against our database. However, I'm not sure what situation would be the case in which there are multiple relevant consents. That said, as @darobin notes, amp-consent or whatever element gets used should be geographically aware enough to only handle these requests within their locale. The case of an EU user arriving on our AMP page while visiting CA is definitely something that seems impossible to resolve to legal satisfaction without a DB check.

Further, I'm not sure how it would work with amp-script as these two consents are very different and GDPR would likely override CCPA. It seems best to make a single API request that included the Reader ID, and expects a return of GDPR and CCPA consent info and makes a decision based on that?

More than the very specific question of CCPA vs GDPR we should assume more processes that have their own compliance parameters are likely to surface in the US and abroad and any system should be able to support not just two, but many potential consent situations.

---

Additional questions:

Beyond this question: AMP will also need to implement the IAB Consent Framework regardless of the state of consent, we will need to indicate that the CCPA IAB Consent framework is active, that we have displayed a compliant opt-out prompt and if the user has opted-out using their framework. We will also need that framework to explicitly signal that a user is not eligible for the opt-out process when they are outside the CCPA compliance area. The consent string of 1-- is needed on all ad HTTP requests (along with any other HTTP request to a potential user tracking service) so vendors can be sure we are checking for compliance and have determined a user is outside the zone. Further the system for a window level object and postMessage receiver will need to be in place.

This is work is required to be complete by January 1 2020. Our team has also determined this will indeed be the enforcement date.

EDIT: Also: this will need to work in AMP Stories.

[zhouyx]

QQ on the opt-out,
If we go with client side storage. We understand there are two limitation. 1. AMP localStorage is not guaranteed to persisted for a minimum of one year. 2. opt out decision from AMP and non AMP is not synced.

If a one time off sync between localStorage and server storage is provided. That AMP will use the local stored value and then ping server to update the stored value. Is this approach acceptable? Thanks.

[zhouyx]

Another option we have considered is pure iframe based. That the <amp-consent> component will initiate an iframe at page load. Then it's up to the iframe to determine what consent is required, collect consents, merge multiple consents if applicable, and sent AMP runtime signal to block/unblock components.
Good point about AMP story. All solutions don't work with AMP story out of the box. But we will work with the story team once the design has been finalized.

[darobin]

I think that using the locally stored value as a cache for the real value is acceptable, but in that case I would tend to think that it would need to be relatively short-lived (and predictable).

I'm am not sure: what is the value in using an iframe as opposed to calling a remote API with the right parameters and letting it state what is active/inactive? It would seem to be heavier?

[zhouyx]

I think that using the locally stored value as a cache for the real value is acceptable, but in that case I would tend to think that it would need to be relatively short-lived (and predictable).

In this case an API to update the local cached value can be provided. The local cached value can be updated every time the user visit the site, and the updated value will be used the next time the user visit the site again.

I'm am not sure: what is the value in using an iframe as opposed to calling a remote API with the right parameters and letting it state what is active/inactive? It would seem to be heavier?

I agree that an iframe seems to be heavier. I proposed iframe instead of a CORS request because of four reasons.

1. Iframe provides the flexibility. The iframe can handle showing different UIs, opt-in/opt-out user flow.
2. AMP already support iframe solution for consent management platform. In the future, I assume many publishers will work with CMPs to collect their user consents. In such cases, iframe solution is preferred and easier to integrate with .
3. To combine multiple consents together, using an iframe provides publishers with the most flexibility. Also given the iframe is already up, it saves the delay of another potential roundtrip request.
4. To support the IAB consent string and its API client side, we need a service to interpret the consent string in the background. Which an iframe provided by a CMP sounds reasonable.

However, we're only exploring the MVP solution right now. If the local storage one-time off solution is good with everyone, I would still propose the client side solution for MVP. Thank you

[zhouyx]

@AramZS and @darobin Can I please get your feedback on the client side one-time off storage solution. The config will be something like:

 <amp-consent id='ABC' layout='nodisplay'>
      <script type="application/json">{
        "consents": {
          "gdpr": {
            "checkConsentHref": "http://localhost:8000/get-consent-v1",
            "promptUI": "ui1"
          },
          "ccpa": {
            "checkConsentHref": "http://localhost:8000/get-consent-ccpa",
            "promptUI": "ui2",
            "skipPrompt": true
          }
        },
        "policy": {
          "default": {
            "waitFor": {
              "gdpr": {
                "timeout": 2,
                "fallbackAction": "reject"
              },
              "ccpa": {
                "timeout": 0,
                "fallbackAction": "accept"
              }
            }
          }
        },
      }</script>
      <div id="ui1">
        Hi EEA reader, can I collect data?
        <button on="tap:ABC.accept" role="button">Accept</button>
        <button on="tap:ABC.reject" role="button">Reject</button>
        <button on="tap:ABC.dismiss" role="button">Dismiss</button>
      </div>
      <div id="ui2">
        Hi CA user, can I collect data?
        <button on="tap:ABC.reject" role="button">Reject</button>
        <button on="tap:ABC.dismiss" role="button">Dismiss</button>
      </div>
    </amp-consent>

[zhouyx]

An alternative is the iframe solution.

Then it will be pure up to publisher to send AMP with decisions on blocking or non blocking. The data flow will be something like this.

where you would have full control within the iframe.

Please let us know your ideas to the two options. It will help us make the design decision and start implementation asap. Thank you.

[lannka]

@zhouyx thanks for the proposals. Could you explain a bit more about the geo detection for both proposals?

[zhouyx]

Sure. AMP will continue to provide geo detection for EEA through <amp-geo>. However no geo detection to detect California users will be provided for MVP. Publishers need to detect CA users either though the checkConsentHref endpoint, or directly within the iframe based on which proposal is adopted.

[AramZS]

@zhouyx Just to be clear, the checkConsentHref will be called by the users' machine and not the AMP cache? And then we would be responsible for determining if that user, based on the request, is in the CA enforcement area?

Also, would the consent screen be able to trigger at different times for the EU and CA users? EU users need an immediate prompt, but CA users would be expected to trigger the opt-out process.

Would we be able to specify a response to user initiated opt-outs via the API-based approach? (like 'You are not in CA so this does not apply')?

[AramZS]

I wanted to update this thread with our current technical understanding of how we expect the IAB Framework to be implemented. The IAB spec is still a draft, so there may be changes there, but I expect this methodology will be close to the version we have Jan 1. We're building our implementation off this understanding of the framework and we would anticipate that AMP's would work similarly. https://github.com/AramZS/IAB-CCPA-Framework-Implementation-Notes

Please let me know if you spot any issues or errors or places where your understanding of the framework implementation would differ so that we can be aware of that in advance.

[zhouyx]

Just to be clear, the checkConsentHref will be called by the users' machine and not the AMP cache? And then we would be responsible for determining if that user, based on the request, is in the CA enforcement area?

That's correct

Also, would the consent screen be able to trigger at different times for the EU and CA users? EU users need an immediate prompt, but CA users would be expected to trigger the opt-out process.

Yes we have that in mvp scope

Would we be able to specify a response to user initiated opt-outs via the API-based approach? (like 'You are not in CA so this does not apply')?

Details is still under discussion. But we haven't included this design for this requirement yet.

[zhouyx]

We will cover this topic in today's design review that's happening in one hour. #25082
Please attend if you can. Thank you!

[zhouyx]

Thanks for the feedback from the design review. Here's the design doc

We've came down to proposal #1 or #2. Please let us know what's your idea on the MVP design. THank you!

[tobie]

Please also find minutes from the design review here: #25082 (comment).

[zhouyx]

Hi all, Thanks for joining the discussion in this thread and yesterday.

After several round of discussions. The AMP team is leaning towards an iframe based solution for all future regulations support. There were a few good points raised by you

1. Performance impact
   We decided to adopt a stale while revalidate model to use cached consent decision to unblock rendering.
2. Implementation difficulty, especially given the tight deadline
   We understand that migrating to the iframe solution takes time. That's why we are proposing a temporary solution to CCPA. That we will add support to fetch consent decision directly from server. The reason we are able to do this is that there will be no promptUI support from AMP to CCPA. We expect publishers to handle and manage user opt-out on their own site.

I've updated the design doc to include the short-term design. Please let us know if this work for you. We are looking to start implementation right after we get a green light from all of you. Thank you.

[lannka]

@AramZS @darobin
This is the proposed MVP spec. Happy to meet in VC tmr to go through any questions you might have.

[lannka]

We have a cleaner spec out here.

Questions are welcome.

[AramZS]

@lannka is the intent of the sharedData object to make the IAB CCPA uspString (or any other opt-out string) available to vendors? (So we would set uspString: 1YN for example into that object.)

To confirm: we would be sending a accepted message to AMP by default to have data-block-on-consent work properly in an opt-out environment?

It appears from the existing documentation that the dismiss action would block those elements waiting on consent? That is the correct behavior for the EU GDPR approach, but the opposite behavior is desired for the CCPA approach.

I am unclear on how this process would work via user initiation in the case of CCPA. For GDPR we needed a consent wall, but for CCPA we want the user to initiate the opt-out. Is there a way to tell the element to trigger in some other way than on-load? I saw some mention of that in the documentation, but not a clear methodology. (For example, if we needed to design and implement a trigger link for the user to initiate opt-out, I'm not sure how to do that from your document.)

Finally, I do have a concern about assuring that AMP extensions respect the data-block-on-consent parameter and pass that consent information downstream. Does the parameter initiate a check to require getConsentPolicy to execute from inside the element? This is not necessarily a blocker, but it is definitely a nice-to-have.

[lannka]

@AramZS thanks reviewing the spec.

For CCPA integration, you basically need 4 things:

1. re-tag your AMP page with a new amp-consent configuration
2. set up a server endpoint for consent state syncing
3. provide a link on your AMP page to initiate the opt-out flow
4. tag the page with data-block-on-consent

(1) amp-consent config

geoOverride: {
  "us": {
      checkConsentHref: "https://example.com/check-consent",
      consentRequired: "remote"
   }
}

(2) server endpoint
Your checkConsentHref server endpoint should return a response like:

{
  consentRequired: true/false, // depending on if the user is in CA
  consentStateValue: "accepted" / "rejected", // depending on if user has opted out
  consentString: "1YN", // the consent string that is accepted by the vendors used on the page
  expireCache: true/false, // depending on if you want to leverage client cache to improve latency for the next visit.
}

note that sharedData is not designed to carry consent string, as it will never be cached locally. It's to pass along other arbitrary one-time data that are agreed between vendor and publisher.

(3) opt-out flow
you need to add a link on your page that takes user to your own website to do the opt-out.

(4) data-block-on-consent
you will likely tag all amp-ad with this attribute so that their instantiation will be blocked until either a) amp-geo says the user is not in regulated region b) your endpoint returned a response.

Let me know if you have further questions.

[lannka]

FYI, the work is tracked here: #25623
We plan to get all done before 12/3. Should be available in canary for testing in that week.

Given the tight timeline, I suggest who ever want to integrate can independently start the server endpoint work as listed in (2) & (3).

[AramZS]

Our team is good with this approach and we have started to plan implementation.

As a heads up, the IAB has released a finalized version of their spec. I have updated my technical documentation as a result, you can see changes at: https://github.com/AramZS/IAB-CCPA-Framework-Implementation-Notes/pull/1/files

[lannka]

FYI, the new spec has been implemented and deployed to dev channel for testing. Let us know how it works.

There is one known issue, to be fixed soon: #25856 (update: already fixed)

[tobie]

@lannka Awesome. Where/how should folks report issues?

[lannka]

Firing separate issues would be preferred. Unless it's very relevant to this issue.

[baconjulie]

Hi @lannka, we've implemented the steps you've outlined here and were wondering if you see anything wrong if our implementation. If it would be better to open a separate issue, let me know but seemed relevant.

We're testing on the dev channel within canary and our code looks like the following:

Geo Config:

{
  "ISOCountryGroups": {
    "eu": ["fr", ... ],
    "us": ["us"]
  }
}

Consent Config:

{
  consents: {
    "my-consent": {
      promptIfUnknownForGeoGroup: "eu",
      promptUI: "consent-ui",
      geoOverride: {
        us: {
          consentRequired: "remote",
          checkConsentHref: "foobar.com"
        }
     }
   }
  }
}

Our amp-ads are tagged with:
data-block-on-consent="_till_responded"

We do see the AMP consent ui trigger for EU users, however it does not appear that the call is being made to our server endpoint at checkConsentHref for US users. See anything we might be missing? Thanks! cc @megapix

[lannka]

@baconjulie could you try:

{
  consentInstantId: "my-consent";
  promptUI: "consent-ui",
  geoOverride: {
    eu: {
      consentRequired: true
    }
    us: {
      consentRequired: "remote",
      checkConsentHref: "foobar.com"
    }
  }
}

The new syntax is here: https://github.com/ampproject/amphtml/blob/master/extensions/amp-consent/amp-consent.md#consent-configuration

/cc @zhouyx @micajuine-ho

[micajuine-ho]

Hi @baconjulie, just to echo what @lannka said:
the geoOverride key has to be a top level key in the inline config or else it will be lost.

We also recommend migrating your inline config to the newest version outlined in the spec above.

Please let us know if you have any other questions!

[baconjulie]

Hi @micajuine-ho @lannka I just tried this locally and it appears to make the call while still showing the consent ui for EU. Thanks a lot, appreciate your help!

[micajuine-ho]

@baconjulie you will need to override the promptUi for the US geo config:

{
  consentInstanceId: "my-consent";
  promptUI: "consent-ui",
  geoOverride: {
    eu: {
      consentRequired: true
    }
    us: {
      consentRequired: "remote",
      checkConsentHref: "foobar.com",
      promptUI: null
    }
  }
}