Consider shared caching

We've had a lot of discussions about using SRI for shared caching (see https://lists.w3.org/Archives/Public/public-webappsec/2015May/0095.html for example). An explicit issue was filed at w3c/webappsec#504 suggesting a sharedcache attribute to imply that shared caching is OK. We should consider leveraging SRI for more aggressive caching.

I hope this is a reasonable place to comment. (If not please tell me where to go.)

I've been working on content addressing systems for several years. I understand that content addresses, which are "locationless," are inherently in conflict with the same-origin policy, which is location-based.

An additional/alternate solution is for a list of acceptable hashes to be published by the server at a well-known location.

For example, the user agent could request https://example.com/.well-known/sri-list, which would return a plain text file with a list of acceptable hashes, one per line. Hashes on this list would be treated as if they were hosted by the server itself, and thus could be fetched from a shared cache while being treated for all intents and purposes like they were fetched from the server in question.

This does add some complexity both for user agents and for site admins. On the other hand, the security implications are well understood, and wouldn't require new permission logic.

Thanks for your work on SRI.

@mozfreddyb

An interesting idea (although I know many folks who are vehemently against well-known location solutions, but I won't pretend to fully grasp why). If implemented, though, it would still require a round trip to get .well-known/sri-list, right? Which seems to lose a lot of the benefit of these acting as libraries.

Another suggestion, that I think I heard somewhere, is, if the page includes a CSP, only use an x-origin cache for an integrity attribute resource if the CSP includes the integrity value in the script-hash whitelist. I think this would address @mozfreddyb's concerns listed in Synzvato/decentraleyes#26, but I haven't thought too hard about it. On the other hand, it also starts to look really weird and complicated :-/

Also, these solutions don't address timing attacks with x-origin caches. Although, as a side not, someone recently pointed out to me that history timing attacks in this case are probably not too concerning from a security perspective since it's a "one-shot" timing attack. That is, the resource is definitively loaded after the attack happens, so you can't attempt the timing again, and that makes the timing attack much more difficult to pull off, since timing attacks usually rely on repeated measurement.

Using a script-hash whitelist in the HTTP headers (as part of CSP or separately) is better for a small number of hashes, since it doesn't require an extra round trip. Using a well-known list is better for a large number of hashes, since it can be cached for a long time.

I agree that well-known locations are ugly. Although it works for /robots.txt and /favicon.ico, there is a high cost for introducing new ones.

The privacy problem is worse than timing attacks: if you control the server, you can tell that no request is ever made. This seems insurmountable for cross-origin caching.

Perhaps the gulf between hashes and locations is too large to span. For true content-addressing systems (like what I'm working on), my preference is to treat all hashes as a single origin (so they can't reference or be referenced by location-based resources).

Thanks for your quick reply!

I'd be slightly more interested in blessing the hashes for cross-origin caches by mentioning in the CSP. .well-known would add another roundtrip. I'm not sure if that's going to impact hamper the performance benefit that we wanted in the first place.

The idea to separate hashed resources into their own origin is interesting, but I don't feel comfortable drilling holes that deep into the existing weirdness of origins.

To be clear, giving hashes their own origin only makes sense if you are loading top-level resources by hash. In that case, you can give access to all other hashes, but prohibit access to ordinary URLs. But that is a long way off for any web browsers and far from the scope of SRI.

@hillbrad

For the record, @hillbrad wrote a great document outlining the privacy and security risks of shared caching: https://hillbrad.github.io/sri-addressable-caching/sri-addressable-caching.html

That document doesn't appear to consider an opt-in approach. While this would reduce the number of people who do it it could be quite useful.

<script src=jquery.js integrity="..." public/>

This tag should only be put on scripts for which timing is not an issue. Of course deciding what is pubic is now the responsibility of the website. However since the benefit would be negligible for anything that is website specific this might be pretty clear. For example loading a script specific to my site has a single URL anyways, so I may as well not put public otherwise malicious sites can figure out who has been to my site recently even though I don't get any benefit from the content-addressed cache. However if I am including jQuery there will be a benefit because there are many different copies on the internet and at the same time it means that knowing whether a user has jQuery in their cache is much less identifying.

That being said if FF had a way to turn this on now I would enable it, I don't see the privacy hit to be large and the performance would be nice to have.

If I want to use the presence of my script in a shared cache to track you illicitly, I will deliberately set the public flag, even if the content isn't actually public.

…

On Mon, Oct 31, 2016 at 3:06 PM Kevin Cox ***@***.***> wrote: That document doesn't appear to consider an opt-in approach. While this would reduce the number of people who do it it could be quite useful. <script src=jquery.js integrity="..." public/> This tag should only be put on scripts for which timing is not an issue. Of course deciding what is pubic is now the responsibility of the website. However since the benefit would be negligible for anything that is website specific this might be pretty clear. For example loading a script specific to my site has a single URL anyways, so I may as well not put public otherwise malicious sites can figure out who has been to my site recently even though I don't get any benefit from the content-addressed cache. However if I am including jQuery there will be a benefit because there are many different copies on the internet and at the same time it means that knowing whether a user has jQuery in their cache is much less identifying. That being said if FF had a way to turn this on now I would enable it, I don't see the privacy hit to be large and the performance would be nice to have. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACFbcMQKkDaic1pBKylEUbpeHMoE2GLOks5q5mZbgaJpZM4G5Tap> .

On 21/12/16 01:07, Brad Hill wrote: If I want to use the presence of my script in a shared cache to track you illicitly, I will deliberately set the public flag, even if the content isn't actually public.

If you want to track me and you control both origins you want to track me from you can just use the same URL and you get a cookie which is better tracking and works today. This is about preventing a third-party site having a script with the same hash as for example a script on Facebook, then they can tell if you have been to facebook "recently". However since fb hosts the script they won't set it as "public" and so it won't be a problem. I don't understand what threat you are trying to protect against.

A "public" flag seems like a good solution to me. It seems to encapsulate both the benefits and the drawbacks of shared caching. It says, "yes, you can share files publicly, but that means anyone can see them."

That said, if it's opt-in, there's the question of how many sites would actually use it, and whether it's worth the trouble. Especially if it has to be set in HTML, rather than say by CDNs automatically. Maybe it would work better as an HTTP header?

Setting in the HTML doesn't seem to be a big problem. If large CDN providers include this in their example script/style tags then sites will copy and paste support for this. A similar approach is currently being used for SRI and although it's not as fast as I'd like, usage will slowly grow. Sites that are also looking for those extra performance boosts would be keen to implement it.

The idea of a public header (or even another key in Cache-Control) sounds quite interesting and elegant, however I think it would make it more difficult to use as one significant use case of this is to let each site to point to their own copy of a script, rather then a centrally hosted one. This means that each site would have to add headers to some of their scripts rather then just a modification in HTML. Not that either is a huge barrier but often static site hosting makes it difficult to set headers especially for a subset of paths.

At the end of the day I have not major objections to either option though.

(I'm assuming that Subresource Integrity Addressable Caching covers all known issues).

The other comment just before my last one has a newish - imho fundamental - blocker. Browsers are already partitioning their cache per first-level site (eventually more granular. Maybe per origin, or per frame-tree).

This issue just turned 7 years old. I'll leave this issue closed because nobody has managed to come up with an idea since.

New issues are cheap. I'm still happy to discuss new and specific proposals - I just currently do not believe those to exist.

The other comment just before my last one has a new_ish_ - imho fundamental - blocker.

Do you mean https://terjanq.github.io/Bug-Bounty/Google/cache-attack-06jd2d2mz2r0/index.html?

In other words, the consequences are worse than initially thought: it's not only a privacy concern, but also a security concern. For example, it enables attackers to use brute-force attacks to guess private data such as a password saved in Google Keep. (Because Google Keep loads different assets depending on whether Google Keep's search returns 0 results, as explained in VI. Google Keep > Vulnerable resource.)

I also think it's a fundamental blocker for small assets such as individual images of an icon library.

That said, I can still see it to be possible to have a shared cache for widespread assets such as React or Angular. Just for the sake of the argument and regardless of feasibility, if websites can declare dependencies in a global manner (e.g. "Google Search" being able to say "I depend on Angular"), then AFAICT this doesn't pose any problems.

Another more interesting example: "Google Keep" can declare that it uses the font "Inter". The interesting thing here is that this doesn't suffer the security issue that I described above, because the dependency is defined globally instead of being defined page-by-page.

As for privacy, it's paramount that only widespread assets (e.g. React, Angular, Inter, ...) can be shared-cached. No code unique to a website/page should ever be shared-cached.

All-in-all I can't see any problems with such high-level goal of enabling websites to globally declare dependencies on widespread assets. Or am I missing something?

While it's challenging to find a concrete technique for implementing this high-level goal (e.g. I'm not sure how a "website" can "globally declare" its "dependency" on an "asset"), I think there is still hope.

Thanks for the discussion, I'm glad if we can bring everyone interested in this on the same page.

This has been investigated in depth by both Google and Mozilla. You're welcome to try again, but the bar at this point is indeed a concrete proposal. This was essentially only found to work if you bundle the libraries with the browser, which creates all kinds of ecosystem problems.

@mozfreddyb

@mozfreddyb I’m sorry, I’m confused. From latest to earliest of your comments, there is the one I’m replying to, then a procedural one, then one that links to Brad Hill’s writeup (which @brillout already mentioned, although without addressing the cross-origin laundering issue—the mention of Want-Digest upthread does that), then one that corrects a previous commenter, says security is the problem, and otherwise amounts to “browsers are switching to separated caches” (which is the one you linked, and it’s true, but I can’t see how it constitutes a “newish blocker”—or did you indeed mean to refer to cache probing with error events as a new security-relevant point?).

@alexshpilkin

@alexshpilkin

@mozfreddyb I’m sorry, I’m confused.

Fair enough, I agree that I haven't been super clear throghout this thread and that I did not re-read through all of it for every comment I submitted.

did you indeed mean to refer to cache probing with error events as a new security-relevant point?).

By "new-ish" I meant "not captured in Brad Hill's doc". Does that answer your question?

-- @annevk said:

This has been investigated in depth by both Google and Mozilla. You're welcome to try again, but the bar at this point is indeed a concrete proposal. This was essentially only found to work if you bundle the libraries with the browser, which creates all kinds of ecosystem problems.

Indeed, the bar is "come up with a proposal that addresses all of these issues" AND either avoids bundling a library into the browser - or does it but then address the (imho) significant ecosystem issues that Anne mentioned. For those new to this, I also found Alex Russel's blog post Cache and Prizes a decent summary which involves the concerns with bundling.

To be extra clear, it's not my intention to be gatekeeping or blocking any progress here. I just want to share what we've discussed and considered, because we thought about it for a very long time. All in all, it could be nice if it was solved well.

For new, concrete proposals please open up a new issue.

I'm glad to hear that there is still interest.

You're welcome to try again, but the bar at this point is indeed a concrete proposal.
For new, concrete proposals please open up a new issue.

Sounds good. Challenge accepted!

I'll be mulling over all of this in the next coming days. I've a couple of design ideas already.

I'll report back.

Thanks for giving me the opportunity to (maybe) make a dent here. I'd be honored.

Alex Russel's blog post Cache and Prizes a decent summary which involves the concerns with bundling

Yes, I agree. As an "underdog OSS developer" (I'm the author of vite-plugin-ssr), I'm particularly attached to foster innovation.

a shared cache would suddenly create disincentives to adopt anything but the last generation of "winner" libraries

That's an inherent drawback of any shared cache, but, considering other aspects, I actually see a shared cache to be a net win for innovation. (I'll elaborate if I manage to find a viable design.)

flat distribution of use among their top versions, the breadth of a shared cache will be much smaller than folks anticipate.

Yes, it's on my radar as well. While I doubt we can do much about the flat distribution aspect (since "don't break the web" is Job #1 for browsers), maybe we can do something about reducing disk usage.

Browser teams aggressively track and manage browser download size

Yes. I strongly believe a shared cache shouldn't be included in the initial browser download. (Or it should include very few things that have like a 99% chance of being downloaded upon the first few websites the user visits.) A shared cached should grow organically, otherwise it becomes a governance mess.

The following design (tentatively) addresses all timing attack concerns (regarding both privacy and security).

If the feedback is positive, we can create a proper RFC in a new GitHub ticket.

The Assets Manifest

A website https://my-domain.com can define an "assets manifest" at https://my-domain.com/assets.json declaring assets for the entire domain my-domain.com.

// https://my-domain.com/assets.json
{
  "assets": {
    "react": {
      "src": "https://some-cdn.com/react/18.2.0",
      "integrity": "sha384-oqVuAfXRKap7fdgcCY5uykM6+R9GqQ8K/uxy9rx7HNQlGYl1kPzQho1wx4JwY8wC"
      "type": "module"
    }
  }
}

<html>
  <head>
    <script name="react"></script>
  </head>
<html>

Having shared assets defined on a per-domain fashion solves the problem of timing attacks determining the user's activity on a given website. (Assets defined in /assets.json are loaded regardless of which page the users visits. Thererefore, the shared cache is completely decoupled from the user activity on a given website.)

Protect user privacy

In order to protect users from timing attacks retrieving the user's browsing history, the shared cache should behave as in the following example.

User installs Firefox (the shared cache is empty).
User visits facebook.com which defines facebook.com/assets.json that has the entry https://some-cdn.com/react/18.2.0.

The browser loads react@18.2.0 and adds it to the local cache of facebook.com but doesn't add it to the shared cached (yet).
User visits netflix.com which defines netflix.com/assets.json that also has the entry https://some-cdn.com/react/18.2.0.

The browser loads react@18.2.0 again and adds it to both the local cache and the shared cache. The idea here is that, at this point, the browser knows that react@18.2.0 is being used by two different domains (facebook.com and netflix.com). This doesn't seem like a sufficient privacy protection at first but I'll elaborate in a moment why it actually is.
User visits discord.com, which also defines discord.com/assets.json with the entry https://some-cdn.com/react/18.2.0.

The browser uses the shared cache to get react@18.2.0

While it may seem surprising at first, the guarantee that a resource is added to the shared cache only after the browser knows it's used by two distinct domains is actually enough to protect users from privacy attacks. Let me elaborate.

Let's consider following example:

User installs Firefox (the shared cache and browser history are empty).
User visits google.com which loads Angular + the Inter font. (And defines these in its /assets.json.)
User visits facebook.com which loads React + the Open Sans font. (And defines these in its /assets.json.)
User visits discord.com which loads React + the Inter font. (And defines these in its /assets.json.)

This means that, at this point, all 4 assets (Angular, React, Open Sans font, Inter font) are in the shared cache, which seems like a glaring privacy leak. But it's actually not.

While it's true that the combination React + Inter uniquely identifies discord.com, a malicious website cannot use that fact to determine the user's browsing history. While the malicious website can use timing attacks to determine that the shared cache contains React and Inter, he cannot determine why React and Inter are in the shared cache. Are React and Inter in the shared cache because the user visited discord.com (loading both React + Inter), or because the user visited facebook.com (loading React) and google.com (loading Inter)? The malicious website cannot determine that.

To be on the safe side, I still think that the browser should be slightly be more conservative and add assets to the shared cache only after the user visted n domains that share the same resource, where n is a number we deem appropriate.

Increase shared cache effeciency

For a library https://some-cdn.com/react to be eligible for the shared cache, it needs to serve https://some-cdn.com/react/18.2.0/diff/18.1.0 which returns the diff between 18.2.0 and 18.1.0. Enabling browsers to quickly update React from 18.1.0 to 18.2.0.

Edge platforms such as Cloudflare Workers make it relatively easy to implement a performant https://some-cdn.com/${some-library}/${versionA}/diff/${versionB}.

Conclusion

I expect questions (especially around privacy) that I'm happy to answer.

If, after discussing this we come to the conclusion it's worthwhile to pursue this direction, we can create a proper RFC.

I'm very excited about the impact we may achieve, especially for low-end devices and low-end networks.

I'm very much looking forward to (critical) feedback.

@brillout

@brillout I’m not sure I like seeing that bound to a domain, but I do see two upsides:

This matches the usual deployment of service workers.
Whether you visit a domain can already be found out by your ISP, but not which pages you visit. Therefore this approach prevents leaking information that is properly hidden with HTTPS.

If a set of companies/domains, for example, alibaba.com and tencent.com and so on uses a library like antdesign-custom.js which is uniquely used by only them we could maybe uniquely identify if a user has visited those specific set of domains
So a solution I thought was - How about adding an allowed domains section? The resource antdesign-custom.js would only be cached for those domains only

On that example, I can, across 2 domains (perhaps in an iframe or redirection chain), can do a timing attack to work out that facebook was visited or not (first domain full loads Open Sans, second domain may/may not), which then discloses if google or discord was visited in turn from the prior timing attack on React + Inter.

up to n domains for how ever many n is for the caching to take hold, discloses however many matches there were pre-existing. and this n can be directly recovered too, just use a library no one else uses.

@Summertime

@Summertime n should be counted towards meaningful user activity. For example, a mere HTTP request to facebook.com isn't counted, while the user spending a couple of hours on facebook.com with many mouse/keyboard events clearly counts as a visit.

@arjunindia Yes, we should take protective measures about this. Example of a very aggressive strategy:

Whitelisting of shared-cache CDNs. (I.e. browsers only add resources to the shared cache that are served by a predetermined list of CDNs.)
Shared-cache CDNs clearly communicate that only library authors should add resources.
The first bytes of the resource need to be:
```
DANGER: DON'T EVER WRITE ME WITHOUT READING https://www.w3.org/warning-about-using-the-shared-cache
```
So that website authors who may be tempted to add their assets to the CDN need to prepend that message to their assets and are therefore further warned.
Shared-cache CDNs automatically remove any resource that don't get a massive amount of requests.
When such resource is removed from the CDN, analyze what went wrong: what kind of resource was it? What website used it? Was it a malicious attempt to make a timing attack? Was it successful?

This is very conservative strategy and I don't think we need to go that far, but it shows that it's in the realm of the possible to address the issue.

I'd even argue that such strategy can be made so effective that we can skip the whole n technique (i.e. setting n to 1).

I chose the n technique for my "RFC seed" comment in order to clearly communicate the key and most important insight of the RFC seed: privacy concerns can be considered in isolation on a per resource basis (making the problem much easier to reason about and much easier to address).

The motivation of this RFC seed is to move the problem from the realm of "very unlikely to ever happen" towards the realm of "possible to implement and worth further investigation".

How about adding an allowed domains section? The resource antdesign-custom.js would only be cached for those domains only

I like that idea, although I'm thinking maybe we should discuss it in a separate "RFC extension". I'm inclined to keep the conversation about the RFC's core propositions and, at some point later and if we can establish confidence around the RFC, we extend the scope of the conversation.

Shared-cache CDNs automatically remove any resource that don't get a massive amount of requests.

A weakness here is resources correlating with user interests (A clear-cut example being peer-to-peer libraries). What if a large site starts pinning a version? What if websites around a topic start pinning a version? What's the diversity of requests.

This means that, at this point, all 4 assets (Angular, React, Open Sans font, Inter font) are in the shared cache, which seems like a glaring privacy leak. But it's actually not.

While it's true that the combination React + Inter uniquely identifies discord.com, a malicious website cannot use that fact to determine the user's browsing history. While the malicious website can use timing attacks to determine that the shared cache contains React and Inter, he cannot determine why React and Inter are in the shared cache. Are React and Inter in the shared cache because the user visited discord.com (loading both React + Inter), or because the user visited facebook.com (loading React) and google.com (loading Inter)? The malicious website cannot determine that.

Even when you can't pinpoint a site with certainty, doesn't mean it doesn't leak information. How do you ensure that any entropy gained from the set of resources in cache is lost among the noise? How do you do this for all light, medium, and heavy users of the web, regardless of the individual?

To be on the safe side, I still think that the browser should be slightly be more conservative and add assets to the shared cache only after the user visted n domains that share the same resource, where n is a number we deem appropriate.

Required reading: Differential Privacy primer by minutephysics and the US Census Bureau: https://www.youtube.com/watch?v=pT19VwBAqKA

There’s still the possibility not to automate it, but instead to have a central list of assets provided by the browser — similar to the decentral eyes extension — or maybe just using it: https://decentraleyes.org/

This sidesteps all the privacy issues and still brings large parts of the benefits. The only part it doesn’t provide is automatic inclusion of new versions, so it disincentivizes updates of common libraries.

@AviKav

@AviKav Yes, the shared cache may indeed leak information about the user if it contains a resource that is (almost) only used by one specific kind of websites, e.g. some JavaScript library for peer-to-peer websites such as The Pirate Bay, or some kind of font used primarily in websites of a certain age group (e.g. a Pokemon font).

Even though it's possible to take further protective measures (e.g. the shared-cache CDN should clearly communicate that topic specific libraries shouldn't be included, the CDN should enforce websites to specify their topic — e.g. with schema.org — and accordingly remove resources from the CDN if the resource correlates with a topic, collect further statistics about what ressources is used on what websites, etc.) I think at this point we put too much requirements on the shared-cache CDN.

Ideally a shared-cache CDN shouldn't be too complex.

Instead of adding further requirements to shared-cache CDNs, we can tackle the privacy problem from the other side: how can we reduce the opportunities for a malicious entity to time attack the shared cache?

For example, we can require the browser to use the shared cache only upon a user activity: if the user navigates to a website by a manual action such as moving her physical mouse over a link or by tapping her touch screen, then and only then are resources loaded from the shared cache. This drastically reduces the opportunity for timing attacks.

I'll elaborate more on this later.

@ArneBab The idea of whitelisting resources has already been brought up and has been disregarded so far (I believe rightfully so).

Navigation Menu

Repository navigation

Consider shared caching #22

40 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Consider shared caching #22

Description

Activity

btrask commented on Apr 24, 2016

joelweinberger commented on Apr 25, 2016

btrask commented on Apr 26, 2016

mozfreddyb commented on Apr 26, 2016

btrask commented on Apr 26, 2016

mozfreddyb commented on Oct 18, 2016

kevincox commented on Oct 31, 2016

hillbrad commented on Dec 21, 2016

kevincox commented on Dec 21, 2016

btrask commented on Dec 21, 2016

ScottHelme commented on Dec 22, 2016

kevincox commented on Jan 2, 2017

40 remaining items

mozfreddyb commented on Jan 3, 2023

brillout commented on Jan 3, 2023

annevk commented on Jan 3, 2023

alexshpilkin commented on Jan 3, 2023

mozfreddyb commented on Jan 4, 2023

brillout commented on Jan 4, 2023

brillout commented on Mar 19, 2023

ArneBab commented on Mar 19, 2023

arjunindia commented on Mar 23, 2023

Summertime commented on Mar 23, 2023

brillout commented on Mar 23, 2023

AviKav commented on Mar 23, 2023

ArneBab commented on Mar 24, 2023

brillout commented on Mar 28, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions