Proposal: Perform additional normalization on input URLs to the declarativeNetRequest API

[oliverdunk]

In Chrome's implementation of declarativeNetRequest, when matching a URL pattern, we use the GURL form of an input URL to check against the underlying URL Pattern Index. The GURL library performs some standardized canonicalization on the raw URL string such as lowercasing the hostname and percent-encoding/decoding certain characters.

We’re considering doing additional normalization on input URLs before checking the index to make it simpler for rulesets to cover variations that should almost always serve the same content - namely decoding all characters except those that would be re-encoded by GURL (such as spaces) and removing trailing dots in fully-qualified domain names.

For example:

https://example.com./abc/8971e618a80dbb21327ae893560a085f.js
https://example.com/abc%2f8971e618a80dbb21327ae893560a085f.js

Would map to:

https://example.com/abc/8971e618a80dbb21327ae893560a085f.js

We would like to propose performing the above operations on all input URLs before matching against a URL pattern.

[Rob--W]

FYI: Firefox's DNR implementation of urlFilter currently passes all tests that existed for Chrome, imported at: https://searchfox.org/mozilla-central/rev/80ae03d93e3fd5769b16f37719b610e359f8fc62/toolkit/components/extensions/test/xpcshell/test_ext_dnr_urlFilter.js#412-1159

During the meeting we covered FQDN matching, but not as much on the other normalization. Could you provide more details on that?

A concern that I have with attempts to normalize is that it may become infeasible for extension developers to target specific requests. Even if https://example.com/abc%252f and https://example.com/abc/2f are handled the same on some/most servers, there may also be servers that treat them differently.

In DNR, there is also a requestDomains property, where we specify that the format should be a canonical domain, which allows for easier matching. In the case of domains, there is hardly any room for ambiguity: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/declarativeNetRequest/RuleCondition#canonical_domain

[mverde]

To be more specific with what we are proposing:

We propose to treat FQDNs as equivalent to domains with no trailing dot when doing DNR rule matching. Currently Chrome does not do this.

We also propose some changes to how percent-encoded characters are handled when doing DNR rule matching. Current Chrome behavior:

• Scheme: percent-encoded characters are considered invalid
• Host:
  • unescaped: alphanumeric + - . [ ] _
  • escaped: everything else
• Path:
  • unescaped: alphanumeric - @ _ ~
  • escaped: 0x00-0x1f space " # < > ? \ ^ ` { | } 0x7f-0xff, non-ASCII Unicode characters
  • unchanged: ! & ' ( ) * + , / : ; = [ ]
• Query:
  • escaped: 0x00-0x1f space # 0x7f, non-ASCII Unicode characters
  • unchanged: all other ASCII characters (no unescaping)
• Fragment:
  • escaped: 0x00-0x1f space " < > ` 0x7f, non-ASCII Unicode characters
  • unchanged: all other ASCII characters (no unescaping)

The proposed change is to unescape everything in the unchanged categories above. The idea is that developers would only have to write one rule that follows the new format to cover all escaped characters variations of valid URLs that would otherwise match the rule, with the caveat that Rob pointed out.

Admittedly this did not originate from any particular request from developers; more that this is something we are considering doing for other internal features that share a call stack with DNR and could imagine it being useful externally. Alternatives:

• If this is considered to not have enough benefit to be worth the effort, we could simply withdraw the proposal.
• If doing this by default is too risky, we could make it optional in the DNR API.

[oliverdunk]

I've been working with @mverde to gather data and understand the tradeoffs here in more detail. To share a few examples I've come up with...

Example 1 (Fully Qualified Domain Names)

Assume a rule like ://example.com/test.txt

Currently, this would match https://example.com/test.txt (probably good) but wouldn't match https://example.com./test.txt (potentially requiring a separate rule to cover).

If we make a change to strip the trailing dot from the input URL before matching, both URLs will be matched by default.

There are two trade offs:

• Theoretically, the server behind https://example.com and https://example.com. could be set up to return different responses for each. I have no idea if that ever happens in practice, but it's worth acknowledging.
• Rules like ://a.ads. in EasyList currently match https://a.ads./example.txt. This would not be the case after this change. However, those rules would still match something like https://a.ads.com/example.txt.

Example 2 (Percent Encoding)

Assume a rule (real, from EasyPrivacy) like /cdn-cgi/pe/bag2?*static.getclicky.com%2Fjs

Currently, this matches /cdn-cgi/pe/bag2?static.getclicky.com%2Fjs but not /cdn-cgi/pe/bag2?static.getclicky.com/js.

If we made this change, neither would match. However, you could update the rule to /cdn-cgi/pe/bag2?*static.getclicky.com/js and it would start matching both.

If any users of the DNR APIs have thoughts here we would really appreciate them :)

[oliverdunk]

@ameshkov @105th I know you are quite active here - do you have any thoughts on this?

I looked through AdGuard's filter lists to identify some examples of rules that may be impacted:

• Many URLs including lots from https://gitlab.com/malware-filter/. These contain percent encoded characters that would be decoded, and so the rules would need to be changed to continue matching. That said, these are often long and with very specific query parameters, which makes me wonder how effective the rules are today.
• Rules like http://www.ads.. This would no longer match http://www.ads./example.txt, but I assume this probably isn't the goal.

To be clear, we're only considering these changes because we think they might be helpful. If the overall sentiment is that they aren't, we will either not go ahead or make it opt-in :)