[Updated October 2020] webRequestBlocking deprecation by Google (Manifest v3)

[Parasrah]

Updates

Latest update (September 4)

Mozilla made a statement on Manifest v3, see #17268 (comment)

Previous updates

View

May 31

Blocking capabilities of webRequest are still being deprecated, see uBlockOrigin/uBlock-issues#338 (comment) (original post deleted) and commentary by @gorhill: uBlockOrigin/uBlock-issues#338 (comment).

Thanks @Giltyhub for update!

Statements made by other browser vendors:

Mozilla

Content blocking: We have no immediate plans to remove blocking webRequest and are working with add-on developers to gain a better understanding of how they use the APIs in question to help determine how to best support them.

[source: Mozilla blog]

Vivaldi

Once the change is introduced to Chromium, believe me when I say that there are many, many possible scenarios.

Restoring the API could be one of them. We’ve restored functionality before.

If the API is removed altogether and no decent alternative is implemented, we might look into creating a limited extensions store.

[source: Vivaldi blog]

Brave

Brendan Eich's answer to ZDNet's inquiry:

To respond on the declarativeWebRequest change (restricting webRequest in full behind an enterprise policy screen), we will continue to support webRequest for all extensions in Brave.

[source: ZDNet (archive.is link)]

Microsoft

None yet.

If you know others, please post a comment in this issue or edit it if you are able to.

Original text by @Parasrah

Hi, I've recently read a string of reports about the chromium project moving to a new API called declarativeNetRequest, and removing the capabilities of the webRequest API to block or redirect requests. From what I understand declarativeNetRequest will still allow blocking and redirect rules, but it will not be as flexible as webRequest in that you will essentially be providing a fairly simple rule-set and will no longer be able to react to a request programatically. They claim this is to increase the privacy for their users and to make redirects/blocking more efficient, although it may also conveniently (for them) break the functionality of many ad-blockers (apparently according to their authors in the reports I've read, I could not find a source for this).

I was wondering how this will affect HTTPS Everywhere (I know you make extensive use of the webRequest API), and if the extension will still be able to function in Chromium. I've also seen speculation that the number of rule sets permitted with declarativeNetRequest would be limited, which would obviously be a problem for this extension if the limit was too low.

Draft of Manifest V3 for chrome extensions

PDF (for those who would like to avoid google docs)

declarativeNetRequest API

Thanks for all the work the team has put in to enhance the security of their users, hopefully there is a way around these changes

[Parasrah]

Update: I see where the speculation on the maximum number of rules came from, the api docs mention it will be 30,000

[zoracon]

@Parasrah Thank you for bringing this to attention. We had some discussion yesterday about how this impacts the extension as the Manifest proposal currently stands. The main concerns are around:

DNR's methods of handling rules are static

• Which means we can't use it to redirect http://example.com/foo to https://example.com/foo
• Rules handling would be statically handled and not programmatically as mentioned above
• The secure cookie feature would be interrupted
• With the static rules.json file, this could mean having to do a full extension update rather than just updating the rules separately like it is handled now.
• Rules are capped at 30K in DNR, reducing our coverage significantly

Service Workers

Transference from background scripts to service workers could potentially allow sites to have more control over the extension's upgrade actions

Missing features in DNR

Redirect looping handling and other features are not necessarily addressed for replacement in declarativeNetRequest are also in question. Error handling is almost nonexistent.

Conclusion

Overall the move to be more static with declarativeNetRequest would take out most of our dynamic functionality - Ultimately blocking the extension's capabilities

There is already bug filed here with lengthy discussion: https://bugs.chromium.org/p/chromium/issues/detail?id=896897&desc=2

This conversation is also being brought up with other privacy oriented extensions:
uBlockOrigin/uBlock-issues#338

[Parasrah]

Thanks for the response, glad to hear this wasn't a surprise and there have been talks internally. Would it be possible to update this issue when a decision has been reached by the team as to next steps?

[zoracon]

@Parasrah yea we will update this as a consensus is made on what to do programmatically. Right now the discussion is still young among us and other privacy extensions. There is a working group having discussion and discourse being had with the proper channels.

Since this is a draft the possibility for changes remain and we are hoping to extend the future of the extension's current capabilities if we clearly outline early on what issues this declarativeNetRequest API currently causes as it stands.

[Parasrah]

Thanks, I appreciate it, and will keep an eye on the various conversations

[pipboy96]

Related: EFForg/privacybadger#2273.

[pipboy96]

Someone needs to pin this issue like the Privacy Badger one.

[zoracon]

Linked is the document I attached to the Google Group discussing Manifest V3. The general thought process is to request functionality that is missing for the declarativeNetRequest API and breaks HTTPSE as it currently stands. Will be on stand by for further responses from the group and watching out for any changes in the dNR API.

https://docs.google.com/document/d/1vzSyzrrjOJrUavfOWC5FC2PNwcBOfN2WkK_HcTdTacw/edit?usp=sharing

[pipboy96]

Related: ungoogled-software/ungoogled-chromium#706.

[zoracon]

As an update, set up general asks if declarativeNetRequest were to happen. So far some responses on what can and can't happen, but generally still watching out for written updates to the potential dNR API

Public Conversation here in the Google Group.
https://groups.google.com/a/chromium.org/d/msg/chromium-extensions/veJy9uAwS00/UfJdEm2eBgAJ

[pipboy96]

@zoracon @Hainish Is there any follow-up to this?

[Hainish]

When was the last edit of this document? Last time I checked we were waiting for a second draft to track what has changed.

[zoracon]

Right now we are still waiting for the second draft of Manifest V3. The last time the document was updated was November 18th, 2018.

Public conversation and any further questions can be posted to the public Google group.
https://groups.google.com/a/chromium.org/d/msg/chromium-extensions/veJy9uAwS00/UfJdEm2eBgAJ

But unfortunately we are stuck in the position of waiting on changes to the draft

[pipboy96]

@zoracon Any updates?

[zoracon]

@zoracon Any updates?

Looking at the docs for declarativeNetRequest, they seemed to have added some changes involving dynamic rules and redirect. The Google Group discussion so far came back with with statement:

That said, the extensions team is currently working on a "developer preview" of the MV3 changes. The intent is to implement most of the major changes to the platform so that extension developers can start transitioning and sharing feedback about the process. We don't have an exact date for this, but we're hoping to ship it in the next few months. - Simeon - @dotproto, Extensions Developer Advocate

Right now, the best action for a HTTPS Everywhere volunteer, user, enthusiast would be to look into jumping into the discussion in the public Google Group discussing these changes. On our end, we are discussing internally what next steps look like and doing our best to create a plan.

[pipboy96]

Removed chrome and firefox labels since they are the only browsers we support anyway.

[zoracon]

Update: 12-17
After much back and forth between different privacy extensions and the Chrome team. They have rolled out Manifest V3 into Canary
https://groups.google.com/a/chromium.org/forum/#!msg/chromium-extensions/hG6ymUx7NoQ/TiAe0gI5AQAJ

First steps:

• Going to load up HTTPS Everywhere in Canary and see what breaks. (Likely everything :( )

- [ ] Next I'm going to move faster to clean up rulesets and run a fetch test as soon as possible
- [ ] Once that is done, I am going to package trivialized rulesets for Chrome (because at this point, the more complicated rulesets will just slow the process down of migration to V3)

• Internally we are working on messaging on how we will communicate this Chrome version
• Please allow patience on other features at this time and design implementation will work will be temporarily put on hold until this is sorted out

Unknowns

• Still not sure how Chrome plans on allocating rulesets, only the privacy extensions mainly use this feature. But this new cap could prove to be a very big challenge.
  Update 12-17: The 30K rulecap is still in effect. Will most likely convert over to an "EASE mode by default" functionality.

• Still not sure on the full impact of migrating from background pages to service workers. I'm very concerned on how debugging the extension will be handled now.
  Update 12-17: There is a bug in discussion here with sleep/wake cycles: https://bugs.chromium.org/p/chromium/issues/detail?id=1024211

[pipboy96]

@zoracon Current Canary can still load Manifest V2 extensions it seems.

[zoracon]

@zoracon Current Canary can still load Manifest V2 extensions it seems.

@pipboy96 Right, they don't specify when V2 will hit EoL. To test V3, just have to change the manifest.json file to "manifest_version": 3. Of course, HTTPS Everywhere simply doesn't install in it's current state. I'll outline more tasks once I read through the migration guide.

[zoracon]

Just a quick update with where we are with this

MV3 is still highly unstable and I am working up a PoC of what the MV3 extension could look like:

Basically, it would have to run strict HTTPS only to get around the ruleset cap as much as possible.

As nice as strict HTTPS is, the volatile changes on MV3's new declarativeNetRequest API and service worker structure are very difficult and frustrating to test with at the moment. There is a bug filed here:
https://bugs.chromium.org/p/chromium/issues/detail?id=1024211

[pipboy96]

@zoracon Do you mean MV3 version will be pretty much EASE-only?

[pipboy96]

@zoracon What's the status of this right now?

[zoracon]

@pipboy96 Pretty much in the same state as when I commented here:
#17268 (comment)

Other developments include the dNR API adding regex rulesets, ability to load different ruleset objects, and better url filtering. With the latest developments + the API finally being more available for testing, I can continue to see what a HTTPS Everywhere in MV3 looks like.

The major issue that I want to tackle is being able to load all applicable rulesets. Which still doesn't seem possible. Google still unilaterally makes changes to the API without much notice. With that said I am working with a similar extension to see what issues we come across collectively. The bigger voice we have, changes do seem to happen in response to our critiques that make the API more tolerable.

However, there are still major concerns on their timeline and when exactly will MV2 will be EOL. Which is still unknown. Hopefully we get a better sense of the timeline, but until then we are just testing changes and trying to adapt while keeping in mind to advocate for privacy extensions to survive MV3.

[zoracon]

Cross referencing my comment made here in #18069 (comment)

"To add to this, MV3 (#17268) is not currently prioritized. The plan is to port to MV3 once they set an actual EoL on MV2. But in order to be prepared, we have to breakdown what we currently have into more modular and compartmentalized code."

With that said, I have created a project that will keep track of the components needed to port successfully and refactor existing code

Edit: Looks like MV2 may be EoL around end of next year. So we can focus on getting the extension in shape for the rest of the year, and we can target January 2021 as the beginning of the port over to MV3 work.