Embedded WebExtension #11760

Hainish · 2017-08-08T21:51:13Z

This PR, superseding #11639 and resolving #10007, migrates settings from the legacy XPCOM codebase to a new, Embedded WebExtensions codebase. This includes:

High-level preferences: Whether HTTPS Everywhere is disabled, whether HTTP requests are allowed, if a counter should be displayed
Which rulesets are manually enabled/disabled
Any custom rulesets which are housed in HTTPSEverywhereCustomUserRules

In addition, this includes various fixes to the functionality of the WebExtensions code to make it work gracefully in Firefox, cherry-picked from the webextensions branch.

Embedded webextensions will be deprecated as of Firefox 57 as well, so this is merely an intermediate step to enable us to have users' settings migrated.

Additionally, this includes a new options ui to allow users to import a json file with settings. Since this replaces #11639, there will be no way for users to export previous settings yet. This won't be a problem, since Embedded WebExtensions are able to transfer settings automatically via the runtime.sendMessage API.

The text was updated successfully, but these errors were encountered:

extension.getBackgroundPage(), see https://bugzilla.mozilla.org/show_bug.cgi?id=1329304 Instead, we're using runtime.sendMessage() to pass messages between the popup and background page.

… to support FF

required due to use of async functions.

…gs migration

Hainish · 2017-08-09T00:16:45Z

In order to maximize the window of time users are given to restart their browsers and have their settings migrated (#11639 (comment)) I'd like to merge this in as soon as possible, and issue a release of the Embedded WebExtension early next week.

ghost · 2017-08-09T00:37:55Z

@Hainish I hope you didn't sacrifice testing to make this come out earlier?

Hainish · 2017-08-09T01:41:37Z

@koops76 I've been testing all along, but there are bound to be edge cases. Any beta-testers that that want to install this are welcome to, but this combines the work over many months of enumerating bugs in the Firefox WebExtension code and pushing out fixes. What's new in this PR is the migration code as well as the XPCOM wrapper.

Hainish · 2017-08-09T02:17:55Z

If you're referring to the testing framework, we have been using the addon-sdk[1] to install a separate extension to test functionality in Firefox. The Jetpack SDK addons will be deprecated just as XPCOM will, so we can no longer use this. I've changed the Firefox tests to use Selenium just as the Chrome tests do. This means we're confined to the bounds of the Selenium API, and currently our coverage is limited. Building this out is a future task.

…

On August 8, 2017 5:37:58 PM PDT, koops76 ***@***.***> wrote: @Hainish I hope you didn't sacrifice testing to make this come out earlier? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #11760 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

jsha · 2017-08-09T05:16:38Z

In addition, this includes various fixes to the functionality of the WebExtensions code to make it work gracefully in Firefox,

These should be in a separate PR.

jsha · 2017-08-09T05:21:18Z

I've changed the Firefox tests to use Selenium just as the Chrome tests do.

This should also be a separate PR.

Bisaloo · 2017-08-09T05:44:24Z

@Hainish, I'm sorry, I'm not sure how to test this.

I checked out to your branch and ran bash makexpi.sh which exited just fine but Firefox is telling me "This add-on could not be installed because it appears to be corrupt."

What am I doing wrong?

ghost · 2017-08-09T06:15:28Z

@Bisaloo Probably a signature issue.

gloomy-ghost · 2017-08-09T08:58:53Z

@Bisaloo You can disable the signature checking by toggling xpinstall.signatures.required, however this option only works for Nightly.

J0WI · 2017-08-09T17:04:22Z

You can disable the signature checking by toggling xpinstall.signatures.required, however this option only works for Nightly.

You should use an old Nightly Build for this, because they are already removing XPCOM support from current Nightlies (57.0a1).

jsha

Some initial comments, haven't reviewed everything yet.

jsha · 2017-08-09T19:01:11Z

chromium/background.js

+
+// Send a message to the embedded webextension bootstrap.js to get settings to import
+chrome.runtime.sendMessage("import-legacy-data", function(settings){
+  import_settings(settings);


You can just provide import_settings directly as the callback parameter to sendMessage, right?

Fixed in #11792

jsha · 2017-08-09T19:01:11Z

chromium/background.js

+});
+
+/**
+ * Enable switch planner for specific tab


Incorrect comment

Fixed in #11792

jsha · 2017-08-09T19:01:11Z

chromium/background.js

+      for(let ruleset of settings.custom_rulesets){
+        all_rules.addFromXml((new DOMParser()).parseFromString(ruleset, 'text/xml'));
+      }
+      storage.set({"legacy_custom_rulesets": settings.custom_rulesets}, item => {


Similarly, no need to define a closure here, just pass resolve as the callback.

Why store this under "legacy_custom_rulesets" rather than store the rulesets in the same way as the other custom rulesets? That way we could just call the appropriate function to load all custom rulesets when we're done, eliminating a possible edge case.

@jsha the other custom rulesets are specially formatted objects that do not allow specifying, for instance, exclusions. Here is one example:

[{"host":"example.com","redirectTo":"https://example.com/","urlMatcher":"^http://example\\.com/"}]

This was never in XML format, since it is added via the WebExtensions UI. From that format, a RuleSet object is created. This format, notably, does not in its current form support exclusions & securecookies. We'd have to build support for that, as well as code that converts XML to the custom_rule format. I'm not against doing this, but it seems like the job for a future migration.

Removal of closure resolved in #11792

Sounds reasonable on the custom rulesets vs XML thing.

Hainish · 2017-08-09T19:49:53Z

PR split into 3 distinct PRs: #11789, #11791, #11792

ghost · 2017-08-11T04:24:31Z

@J0WI Move to "Done" please.

Hainish · 2017-08-11T04:41:37Z

This PR wasn't merged so I'll just remove it from the board

Hainish added 9 commits Aug 8, 2017

Naive fix for absence of chrome.windows and chrome.tabs (Fennec)

66cd627

Ensure that storage api falls back to local when sync is unavailable

16f2545

Update manifest.json to add Firefox-specific keys

f5188b4

Change author key in manifest.json to be a string

95f62d1

Firefox fix: Private Browsing Mode popups cannot use

a2d1b1d

extension.getBackgroundPage(), see https://bugzilla.mozilla.org/show_bug.cgi?id=1329304 Instead, we're using runtime.sendMessage() to pass messages between the popup and background page.

Do not try to display image if we're not in Chrome browser

2948c3d

Create an additional lookup method for favicons.

00dd74c

Add spacing between checkbox and ruleset name in popup

aba46a8

Firefox: Removing SDK extension tests and generalizing selenium tests…

1993ef3

… to support FF

Hainish mentioned this pull request Aug 8, 2017

WIP: Export settings #11639

Closed

Hainish added this to In Progress in Firefox WebExtension Aug 8, 2017

Hainish force-pushed the embedded-webextension branch from 856656a to d190662 Aug 8, 2017

Hainish added 2 commits Aug 8, 2017

Adding options UI for the ability to import settings. Chrome 55+ is

fdd933f

required due to use of async functions.

Converting the XPCOM extension to an Embedded WebExtension for settin…

48d8d3e

…gs migration

Hainish force-pushed the embedded-webextension branch from d190662 to 48d8d3e Aug 8, 2017

Hainish mentioned this pull request Aug 8, 2017

Strange behaviour for 'Block all unencrypted requests' after browser restart #10478

Closed

Hainish changed the title ~~WIP: Embedded webextension~~ Embedded webextension Aug 9, 2017

Hainish changed the title ~~Embedded webextension~~ Embedded WebExtension Aug 9, 2017

jsha suggested changes Aug 9, 2017

View changes

Hainish closed this Aug 9, 2017

J0WI moved this from In Progress to Done in Firefox WebExtension Aug 9, 2017

J0WI removed this from Done in Firefox WebExtension Aug 9, 2017

J0WI added this to Won't fix in Firefox WebExtension Aug 9, 2017

Hainish removed this from Won't fix in Firefox WebExtension Aug 11, 2017

Hainish deleted the embedded-webextension branch Feb 23, 2018

EFForg / https-everywhere Public

Embedded WebExtension #11760

Embedded WebExtension #11760

Hainish commented Aug 8, 2017

Hainish commented Aug 9, 2017

ghost commented Aug 9, 2017

Hainish commented Aug 9, 2017

Hainish commented Aug 9, 2017

jsha commented Aug 9, 2017

jsha commented Aug 9, 2017

Bisaloo commented Aug 9, 2017

ghost commented Aug 9, 2017

gloomy-ghost commented Aug 9, 2017

J0WI commented Aug 9, 2017

jsha left a comment

jsha Aug 9, 2017

Hainish Aug 9, 2017

jsha Aug 9, 2017

Hainish Aug 9, 2017

jsha Aug 9, 2017

Hainish Aug 9, 2017 •

edited

Hainish Aug 9, 2017

jsha Aug 9, 2017

Hainish commented Aug 9, 2017

ghost commented Aug 11, 2017

Hainish commented Aug 11, 2017

EFForg / https-everywhere Public

Embedded WebExtension #11760

Embedded WebExtension #11760

Conversation

Hainish commented Aug 8, 2017

Hainish commented Aug 9, 2017

ghost commented Aug 9, 2017

Hainish commented Aug 9, 2017

Hainish commented Aug 9, 2017

jsha commented Aug 9, 2017

jsha commented Aug 9, 2017

Bisaloo commented Aug 9, 2017

ghost commented Aug 9, 2017

gloomy-ghost commented Aug 9, 2017

J0WI commented Aug 9, 2017

jsha left a comment

jsha Aug 9, 2017

Hainish Aug 9, 2017

jsha Aug 9, 2017

Hainish Aug 9, 2017

jsha Aug 9, 2017

Hainish Aug 9, 2017 • edited

Hainish Aug 9, 2017

jsha Aug 9, 2017

Hainish commented Aug 9, 2017

ghost commented Aug 11, 2017

Hainish commented Aug 11, 2017

Hainish Aug 9, 2017 •

edited