Portability

[Deathamns]

Main goals

• Encapsulate extension APIs into a common API
• Support Safari
• Provide a build script for packaging extension files

Structure

├─ dist/
│  └─ build         # created when running the build script
│     └─ #.#.#.#    # [version number] contains the built files
├─ doc/             # extension documentation
├─ meta/            # configuration and vendor specific files
│  ├─ crx/          # Chromium related files
│  ├─ safariextz/   # Safari related files
│  ├─ xpi/          # Firefox related files
│  └─ config.json   # extension metadata (name, version, author...)
├─ src/             # source code
├─ tmp/             # temp directory, exists when the building is in progress
├─ tools/           # build scripts
└─ other files...   # README, LICENSE...

vendor API (vAPI)

The following files are responsible for the extension API encapsulation.

/src/js/vapi-appifo.js contains some common informations about the extension (like name, version string, homepage). It can be included where it's needed.
The reason for this is that not every browser provides an easy way to access these informations (Chrome does, Safari doesn't, Firefox gives an asynchronous method).
The content of this file is automatically updated when building.

/src/js/vapi-background.js included only in the background process.
It encloses the "privileged" APIs (like net request, tabs), that can run only in the background process.

/src/js/vapi-common.js it can be included in background process and extension pages (e.g., settings or pop-up), and it holds common APIs that could be used in both places (i18n for example).

/src/js/vapi-client.js this also holds common APIs, but it can be included into extension pages and as content-scripts.
Currently it handles messaging, and contains extra code for specific vendors (if needed - see Safari).

Note that this API is not an extensive framework, only implements code that are needed for the extension to work properly.

The files contain all the vendor specific code. Which part will run is determined by simple browser detection, but when building, only the correct code pieces are included into the corresponding extension files.

Basically these are the only js files which are different across browsers (if we don't count the extra files for Firefox (2) and Safari (1)).

Metadata

These informations are used when building.

In order to keep extension data in one place, instead of updating all the vendor specific files (manifest.json, Info.plist...), the /meta/config.json keeps these common informations (such as name, version, author...), also contains vendor specific informations (extension IDs, path to private key which is used when building).

Installing

The source code can be found under the src directory, and since all the manifest files and locales are not cleared when building (intentional), the extension can be installed for all browsers from this directory without building or configuring anything first.

Chromium based browsers

Go to the Extensions page (chrome://extensions), and drag n' drop the src folder onto the browser window.

Safari

First you need to create a developer account and request a certificate from Apple, which you have to install on your machine. More info.

After you have your certificate installed...

Create a link named uBlock.safariextension for the src directory, open the Extension Builder in Safari, click Add Extension, and select the created link.

In Windows a Directory Junction needs to be created:

mklink /H /D /J uBlock.safariextension src

Building (optional)

Dependecy

• Python3
• Unix enviroment or cygwin for Windows with the following commands available
  • 7z (Chrome, Firefox)
  • openssl (Chrome)
  • xar (packaging for Safari)

Safari needs a few extra steps first, which are listed here (step 1,2,3).

---

The building will:

• update manifest files with the data from /meta/config.json
• generate and copy necessary files for specific browser (e.g., locales)
• separate browser specific code from vapi-*.js files
• create update-files
• pack the code into installable or publishable extension files

The result files are placed into /dist/build/#.#.#.# (version) folder. If the same folder already exists, first it will be removed with its content.

To build everything, run the following command:

$ python tools/build.py

To refresh only the language and manifest files (note the optional meta argument):

$ python tools/build.py meta

[Deathamns]

Sorry for the quality of the description, my English is preventing me to do better.

[gorhill]

Ok so finally I'm trying to go through the portability code to regain back the mental picture of the project. I will go through and review carefully all the files.

First thing, one thing I see right away is that I would like that each platform get its custom vapi-*.js files, regardless whether they are very similar or not. Benefit is to eliminate the large if-else code inside it -- something I highly dislike, and also since I can't test on Safari, it is a good thing to restrict changes to one platform, so that I don't have to test other platforms.

My question, where should the platform-specific versions of vapi-*.js be? In ./meta/{vendor}?

There is a Settings.plist in the root folder of the project. Can I removed it, since there is a copy in ./meta/safariextz?

[Deathamns]

You could simply just create clones of that file and name them vapi-client-vendor_ext.js.
But, whatever you choose, separating those vapi files can cause a few problems.
Since each of them will have a unique URI (for example meta/vendor_ext/vapi-client.js if they go to meta, or src/js/vapi-client-vendor_ext.js if they stay in src/js), the possibility for installing the extension from the same src folder for different vendors would break, because all the files that reference the vapi files must have their own versions too, which you could achieve by building for a specific vendor (after that the src folder could be installed onto only one browser, until it's not build for another one) or by duplicating those files (so there would be dashboard-crx.html and dashboard-xpi.html files, which would reference the corresponding vapi file).

meta files are copied/updated to the src folder when the build.py is called, and they stay there, so someone can install it immediately from that folder. It just happens that Settings.plist doesn't has variables in it (so it won't differ form the file in meta), but it is the same category as manifest.json or Info.plist.

[gorhill]

Ok, I went ahead and saw your answer after I pushed some of the changes.

separating those vapi files can cause a few problems

With uMatrix, I create a paltform-specific folder in dist, and use the result as a release. I would prefer to do this here too. Installing directly from the src folder is not really a big deal, I could/would never guarantee that it would work anyway, as it does happen that I commit changes while in the middle of larger changes. The releases at https://github.com/gorhill/uBlock/releases is where all working releases will go.

I really want to separate completely the code shared by all platforms versus the code specific to a single platform, even if it means there will be some duplication because one platform is somewhat close to another.

Also, I would prefer a zipped release for Chromium-based browser. My understanding is that crx is no longer the preferred way to go, as the extension will be force-removed from Chrome/Windows as per the new Chrome policy which forces extensions to be installed from the store. Manually installing through zip at most results in an annoying warning at start for Chrome/Windows users.

[gorhill]

Couple more questions:

What is the reason for having the content of vapi-appinfo.js not inside vapi-common.js?

I see that the port management background page-side now uses closures upon connection. There was care to avoid overhead for connection stuff, in order to perform well for such stress test as the vim-color pages, in which each embedded frame results in a connection to the background page.

I was also using a method to avoid any potential port name collision by having the background page pick the port name, rather than having the client play lottery to pick a port name. I used to use the lottery approach originally, but the thought of some difficult-to-track bug surfacing because of a collision in port name caused me to rewrite to be 100% sure there were no collisions.

Is there a problem in Safari with having the background page rename the port on its side to ensure uniqueness? If there is no problem, then I could put back the code for port naming, and this will then open the door to get rid of the closures when a connection occurs.

[Deathamns]

So, basically you don't want to use the build script I made. In that case, you can even remove it, because it's not tied to anything, and use the script you used before.
You can move Info.plist, Settings.plist, js/sitepatch-safari.js and locales.json to meta/safariextz, and manifest.json to meta/crx from the src folder, if you don't want to see platform specific files there.

Just for the record, the meta folder was intended to hold metadata, but now if you put code and filled manifest files there, then maybe it could be renamed to something more related, like vendor or platform.

vapi-appinfo.js not inside vapi-common.js

Because that file is generated (so only that file was touched) and can be included even as a content script. However, if you know that you won't use as a content script, you can move it to vapi-common.js.

closures

I just tried the latest stable and the dev version of the extension with a recent Chromium build on that vim style-preview site, and I didn't notice any difference. Nor in loading the page, nor in memory usage (I got around 34Mb for each version).

ports

Ports are Chrome specific, so basically I tried to recreate a simple sendMessage for Chrome too, but with ports (since you used those, and saw that you mentioned a crbug about a memory leak - however I always used sendMessage, and never experienced problems with it).
Most websites use just a few frames most, so the generating unique ids should be more than enough.

[gorhill]

I just saw that in vapi-client.js now a single connection is reused for multiple channels -- while before it was one connection per channel. I like this, I like your concept of channels.

So, basically you don't want to use the build script I made

I am not against the idea of build.py, it's just that I have a hard time with the idea of having code for multiple platforms in one single JS file. If they are clearly separated into their own file, this is easier to maintain IMO, as long as we all agree about what goes into vAPI. And this simplifies building as we don't need a parser based on custom markers, and this does simplify my job as maintainer of this project: copying files is easier then writing Python code to parse/extract information. Maybe it will be unavoidable to do so, but surely we don't have to do everything with it.

However, if you know that you won't use as a content script, you can move it to vapi-common.js.

Ok I had in mind vapi-common.js was also for content scripts. I think I understand now: vapi-client.js is for extension pages + content scripts, while vapi-common.js is for background page + extension pages.

Re. overhead: if it can be avoided without overcomplicating, I rather avoid it. sendMessage will create and destroy a port each time a message is send last time I checked (stepping with debugger).

Take Tweetdeck, you will find that uBlock's mutation observer is working hard, causing a lot of back and forth messaging. It makes me feel better to know that whatever overhead could be avoided is actually avoided. Once the code has been written, we can forget about it knowing it can't be pushed further.

It's not because with a quick test we can't notice that there is an overhead, we should not try to avoid it. Just having some ideas of what is going on behind the scene is reason enough for me. Creating two new functions + associated closure for each port connection surely there is a cost, and in this case is avoidable without over-complication of the code.

It's caring about countless if these "unnoticeable" overheads that made this project what it is, I want to stick to this.