Correct / real URLs should be enforced, to avoid breaking adblockers

[pes10k]

Currently there is no enforced relationship between the URL used to look up resources in the package, and where the resource came from online. Consistent URLs are an imperfect, but extremely useful signal for privacy protecting tools (filter lists, adblockers, disconnect, Firefox and Edge built in protections, safe browsing, etc.).

The current proposal would allow for all WebPackage'd sites to circumvent all URL based tools by simply randomizing URLs as a post processing step in amppackager or similar. This could even be done per-request per page. Since URLs are effectively just indexes into the package (and not keys for decision making, caching, etc), they can be changed arbitrarily w/o affecting how the package loads, but preventing the URL-based privacy preserving tools from running.

A (partial) possible solution to the problem is to play a cut-and-choose, commitment-auditing style games with the URLs. At package time, the packager has to make commitments about which URL each resource came from, and the size, shape etc of the resource. These can be made / mixed with the URL of the page being packaged.

The client can then, w/ some probability, audit some number of the URLs in the package. If the commitments fail, deterring counter measures can be taken against the packing origin (e.g. global decaying block list of misbehaving packagers, etc).

[jyasskin]

Because this is an issue where the potential attackers may not have thought of all the attacks we want to defend against, I don't want to discuss this issue in public. I'm going to try to discuss it in https://github.com/WICG/webpackage/security/advisories/GHSA-g5qv-3cw4-38gv instead. Send me an email with an aspect of the problem that isn't yet discussed here in order to be added to that discussion.

[pes10k]

just wanted to check in on this, has anything changed / any updates?

[pes10k]

Copying comments over from the closed PR thread in #573, and editing slightly given the new context

In general, i'm happy to continue discussing point by point above, but lets not loose the forest for the trees. The general claim is that:

1. consistent, descriptive URLs are useful for adblocking (edit, and content blocking in general)
2. this proposal reduces the consistency and descriptiveness of URLs by changing them into arbitrary, opaque indexes into an archive.

Are we disagreeing about either of the above points?

Since rollup got mentioned above, its a perfect example here. Before rollup-and-the-link world, content blocking was ideal; URLs described (both conceptually, and frequently) one resource, and the user agent could reason about each URL independently. Post rollup-world, URLs are less useful (though not useless), since JS URLs now describe (often) many resources, about which its increasingly difficult for the UA to reason individually about. (on going research here, etc). URLs represent multiple interests, the user will often feel differnetly about, but which UA's are (generally) forced into an all or nothing position about.

This proposal does the same thing, but for websites entirely! The UA effectively gets just one URL to reason about (the entire web package), but looses the ability to reason about sub resources. This is very (very!) bad if we intend the web to be an open, transparent, user-first system!

Okie, now, replying to individual points, but eager to not loose site of the above big picture…

@jyasskin

#573 (comment) is wrong about the performance implications

This is not correct. Its partially correct in v8, bc in some cases v8 will defer the parsing of function bodies, but (i) even then there are exceptions, and (ii) I have even less familiarity with how other JS engines do this. I know that, for example, spidermonkey does not not defer parsing in cases where v8 will (e.g. JS in HTML attributes, onclick=X), but I dont have enough information to say in general (and I know even less about JavaScriptCore). But, point is

1. there is in all cases some difference, because there is at lease some additional parsing and executing going on
2. there may be significant difference in other platforms
3. caching makes all this even more different, as platforms may differ on how and when they cache inline script
4. none of this difference hangs on standards defined behavior, and so is not a sound basis for this standard to rely on

@twifkak

can't the site choose not to include the 3p script in the bundle

Sure, a site could choose this, but i'm not sure I follow the point. My point isn't that sites have to evade content blockers in the proposal, its that it gives them new options to circumvent the user's goals / aims / wishes.

As for collisions between bundles and the unbundled web…

Again, im not sure I follow you here. My point is that it'd be simple to change URLs during "bundling" so that they're (i) impossible for content blockers to reason about, and (ii) ensure they don't collide with real world urls. Say, every bundled resource has its URL changed to be a random domain 256 character domain and path.

My example involves changes that would mostly be internal to the CMS, and hence the cost amortized across its customers

Needing to update the large number of existing CMS's seems like a perfect example of why this is difficult for sites! Let alone other costs (loosing cache, in your hash guessing scheme paying an extra network request and on some platform OS thread or process, making static sites unworkable, etc etc etc).

TL;DR as much as possible,

1. Yes URLs can be opaque on the web no2
2. they are never the less still useful (see Google safe browsing, EasyList, disconnect, caching policies etc etc etc)
3. the claim isn't that this proposal does something to fundamentally change URLs, its that it (i) takes something expensive and possible for the server to do now, and makes it free and trivial, and (ii) makes (for packages) the entire package into a single yes / no decision for the UA, where before the UA had far more information and ability to choose / advocate on behalf of the user

[twifkak]

2. this proposal allows reduces the consistency and descriptiveness of URLs by changing them into arbitrary, opaque indexes into an archive.

"Disagreeing" implies 100% confidence to me, so let's just say I'm skeptical of this. I'm certainly counter-arguing the point.

This proposal does the same thing [as rollup]

Except that it establishes distinct boundaries between resources. It may be easier to detect matching JS subresources inside a bundle than inside a rollup, since they are distinct and the publisher has less incentive to mangle them (no need to avoid JS global namespace conflicts).

It could include rollup'd JS payloads that contain a mix of 1p and 3p content, but I don't see how doing so helps evade detection over unbundled rollup.

can't the site choose not to include the 3p script in the bundle

Sure, a site could choose this, but i'm not sure I follow the point. My point isn't that sites have to evade content blockers in the proposal, its that it gives them new options to circumvent the user's goals / aims / wishes.

I was responding to your comment "web bundles give sites a new way of delivering code to users... in a way that has zero additional marginal cost... since the code is already delivered / downloaded as part of the bundle, there is no additional cost to making it an async request vs inlining it".

We're somewhat in subjective space here, but I'd argue that the apples-to-apples comparison is:

• unbundled: link to 3p script vs rollup or inline or 1p mirror
• bundled: link to 3p script vs rollup or inline or 1p mirror or bundle

Regarding bytes delivered over the network from edge server to browser, bundling doesn't appear to change the cost relative to baseline. I haven't thought through bytes at rest, or between various layers of serving hierarchy. I wonder the degree to which such a cost is the limiting factor right now.

It does offer another option for 1p-ifying the script in order to evade detection, but one that doesn't seem to offer the site any reduced marginal cost.

As for collisions between bundles and the unbundled web…

Again, im not sure I follow you here. My point is that it'd be simple to change URLs during "bundling" so that they're (i) impossible for content blockers to reason about, and (ii) ensure they don't collide with real world urls. Say, every bundled resource has its URL changed to be a random domain 256 character domain and path.

I was arguing that it might not be so simple, depending on the circumstances. HTTP cache and ServiceWorker might offer spaces for collision between bundled URLs and unbundled URLs. Thus, making random paths undetectable seems similarly hard in both the unbundled and bundled world.

Your point about random domains is interesting. In order to be undetectable, the random domain+paths have to look real. Given that servers may vary their responses to different requestors, it's impossible to know that a real-looking 3p URL doesn't name a real resource (and that it won't over the length of the bundled resource's lifetime). However, it's probably sufficient to assert no collision on the cache (e.g. double- or triple-) key. A bundle generator need only an avoid-list of 3p URLs the site uses. I'm not sure if this allows an easier implementation than my proposed path randomizer, though.

Needing to update the large number of existing CMS's seems like a perfect example of why this is difficult for sites!

Isn't an update also necessary for adding bundle support?

Let alone other costs (loosing cache, in your hash guessing scheme paying an extra network request and on some platform OS thread or process, making static sites unworkable, etc etc etc).

In these aspects, it would be interesting to compare the costs between bundled and unbundled blocklist-avoidance in more detail.

[pes10k]

It could include rollup'd JS payloads that contain a mix of 1p and 3p content, but I don't see how doing so helps evade detection over unbundled rollup.

There are big differences. You can't roll up 3p scripts (easily), and you can't roll up the other kinds of resources folks might want to block (images, videos, etc etc).

I didn't mean to suggest that this is just like rollup on a technical level; only that it further turns websites into black boxes that UA's can't be selective about / advocate for the user in, and in that way was similar to rollup.

It does offer another option for 1p-ifying the script in order to evade detection, but one that doesn't seem to offer the site any reduced marginal cost.

The difference here is that to get the kind of evasion you can get in a web bundle, you'd need to roll into an existing script, inline the code, or pull it into a 1p URL (that could itself be targeted by filter lists, etc). In a WebBundle world, the bundler has the best option for evading, without having to do the more difficult work (i.e. zero marginal cost).

I'm fine to say small marginal cost if that gets us by this point, but the general point is that sites get new evasion capabilities at-little-to-no-cost.

A bundle generator need only an avoid-list of 3p URLs the site uses. I'm not sure if this allows an easier implementation than my proposed path randomizer, though.

Its easier bc

1. You have to just write the evasion in one place (the bundler) instead of changing ever app on the web!
2. You get the evasion / opaque URL without needing to do network guess games, etc, which can be expensive (think Drupal plus apache!) or straight up impossible (static site generators)

Isn't an update also necessary for adding bundle support?

I can't see why. At least for sites where the bundle content is static (AMP like pages), I have all the information I need to build the bundle just by pointing at an existing site / URL, no changing of CMS needed (you might want to add options for excluding certain domains, resources, etc, but thats all equally easy and do-once-for-the-whole-web)

[twifkak]

Isn't an update also necessary for adding bundle support?

I can't see why. At least for sites where the bundle content is static (AMP like pages), I have all the information I need to build the bundle just by pointing at an existing site / URL, no changing of CMS needed (you might want to add options for excluding certain domains, resources, etc, but thats all equally easy and do-once-for-the-whole-web)

By "I" in your second sentence, who do you mean? If "a distributor" or "the site's CDN", why would they limit such a technique to unsigned bundles? The argument that pulling content into a 1p URL is difficult enough to impede adoption doesn't seem to apply in this case. (edit: Likewise with a "2p" subdomain dedicated to mirroring content of a given 3p.)

The degree to which the HTML is amenable to static analysis seems to affect the feasibility of such an implementation, but not along the bundled-or-not axis.

[pes10k]

in that particular case, i just meant a site maintainer looking to create a web bundle. You were making the argument (if i understood correctly) that it would be the same amount of work to rewrite a CMS to create web bundles, as it would be to rewrite a CMS to do other kinds of URL-based filtering evasion. My point was just that no CSM rewriting would be needed at all to web bundle. I can treat the server side code as a blackbox, poke at it with automation, and create a bundle from the results (i.e. if I can create a record-replay style HAR of the site, i can create a web bundle of the site).

(I also think this is not the right way to think about the comparison, since rewriting the CMS is just one of many things you'd need to change to do filter list evasion: caching, performance concerns, etc etc etc)

[twifkak]

Ah, I see. I lack sufficient awareness of sites and their owners, to judge whether "update my CMS version" or "install an HTTP middlebox" is harder, on average (including auxiliary changes such as you mention, in both cases). I could see it going both ways.

[pes10k]

I don't think its likely useful for us to speculate about which is easier in general, but at the least I think / hope we can agree on:

1. Web packaging would allow new categories of folks to obscure URLs that currently can't, bc they're not in a position to run middleware, like static site generators, people on shared hosting that scrapes down to apache and (S)FTP, people with heavy cache needs because of DDOS or other concerns, etc
2. It would allow them to do so w/o having to pay the costs they'd pay to do something similar on live websites (again caching, handling extra requests due to hash-misses in the scheme you proposed, etc)