Skip to content
Permalink
Browse files
Add experimental mv3 version
This create a separate Chromium extension, named
"uBO Minus (MV3)".

This experimental mv3 version supports only the blocking of
network requests through the declarativeNetRequest API, so as
to abide by the stated MV3 philosophy of not requiring broad
"read/modify data" permission. Accordingly, the extension
should not trigger the warning at installation time:

    Read and change all your data on all websites

The consequences of being permission-less are the following:

- No cosmetic filtering (##)
- No scriptlet injection (##+js)
- No redirect= filters
- No csp= filters
- No removeparam= filters

At this point there is no popup panel or options pages.

The default filterset correspond to the default filterset of
uBO proper:

Listset for 'default':
  https://ublockorigin.github.io/uAssets/filters/badware.txt
  https://ublockorigin.github.io/uAssets/filters/filters.txt
  https://ublockorigin.github.io/uAssets/filters/filters-2020.txt
  https://ublockorigin.github.io/uAssets/filters/filters-2021.txt
  https://ublockorigin.github.io/uAssets/filters/filters-2022.txt
  https://ublockorigin.github.io/uAssets/filters/privacy.txt
  https://ublockorigin.github.io/uAssets/filters/quick-fixes.txt
  https://ublockorigin.github.io/uAssets/filters/resource-abuse.txt
  https://ublockorigin.github.io/uAssets/filters/unbreak.txt
  https://easylist.to/easylist/easylist.txt
  https://easylist.to/easylist/easyprivacy.txt
  https://malware-filter.gitlab.io/malware-filter/urlhaus-filter-online.txt
  https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=1&mimetype=plaintext

The result of the conversion of the filters in all these
filter lists is as follow:

Ruleset size for 'default': 22245
  Good: 21408
  Maybe good (regexes): 127
  redirect-rule= (discarded): 458
  csp= (discarded): 85
  removeparams= (discarded): 22
  Unsupported: 145

The fact that the number of DNR rules are far lower than the
number of network filters reported in uBO comes from the fact
that lists-to-rulesets converter does its best to coallesce
filters into minimal set of rules. Notably, the DNR's
requestDomains condition property allows to create a single
DNR rule out of all pure hostname-based filters.

Regex-based rules are dynamically added at launch time since
they must be validated as valid DNR regexes through
isRegexSupported() API call.

At this point I consider being permission-less the limiting
factor: if broad "read/modify data" permission is to be used,
than there is not much point for an MV3 version over MV2, just
use the MV2 version if you want to benefit all the features
which can't be implemented without broad "read/modify data"
permission.

To locally build the MV3 extension:

    make mv3

Then load the resulting extension directory in the browser
using the "Load unpacked" button.

From now on there will be a uBlock0.mv3.zip package available
in each release.
  • Loading branch information
gorhill committed Sep 6, 2022
1 parent 1def4e7 commit a559f5f2715c58fea4de09330cf3d06194ccc897
Show file tree
Hide file tree
Showing 28 changed files with 1,651 additions and 368 deletions.
@@ -45,6 +45,7 @@ jobs:
tools/make-firefox.sh ${{ steps.release_info.outputs.VERSION }}
tools/make-thunderbird.sh ${{ steps.release_info.outputs.VERSION }}
tools/make-npm.sh ${{ steps.release_info.outputs.VERSION }}
tools/make-mv3.sh all
- name: Upload Chromium package
uses: actions/upload-release-asset@v1
env:
@@ -81,3 +82,12 @@ jobs:
asset_path: dist/build/uBlock0_${{ steps.release_info.outputs.VERSION }}.npm.tgz
asset_name: uBlock0_${{ steps.release_info.outputs.VERSION }}.npm.tgz
asset_content_type: application/octet-stream
- name: Upload Chromium MV3 package
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: dist/build/uBlock0.mv3.zip
asset_name: uBlock0.mv3.zip
asset_content_type: application/octet-stream
@@ -1,7 +1,7 @@
# https://stackoverflow.com/a/6273809
run_options := $(filter-out $@,$(MAKECMDGOALS))

.PHONY: all clean test lint chromium firefox npm dig \
.PHONY: all clean test lint chromium firefox npm dig mv3 \
compare maxcost medcost mincost modifiers record wasm

sources := $(wildcard assets/resources/* src/* src/*/* src/*/*/* src/*/*/*/*)
@@ -52,6 +52,11 @@ dig: dist/build/uBlock0.dig
dig-snfe: dig
cd dist/build/uBlock0.dig && npm run snfe $(run_options)

dist/build/uBlock0.mv3: tools/make-mv3.sh $(sources) $(platform)
tools/make-mv3.sh all

mv3: dist/build/uBlock0.mv3

# Update submodules.
update-submodules:
tools/update-submodules.sh
@@ -37,7 +37,10 @@ vAPI.setTimeout = vAPI.setTimeout || self.setTimeout.bind(self);

vAPI.webextFlavor = {
major: 0,
soup: new Set()
soup: new Set(),
get env() {
return Array.from(this.soup);
}
};

(( ) => {
@@ -0,0 +1,65 @@
'use strict';

import regexRulesets from '/rulesets/regexes.js';

const dnr = chrome.declarativeNetRequest;

dnr.setExtensionActionOptions({ displayActionCountAsBadgeText: true });

(async ( ) => {
const allRules = [];
const toCheck = [];
for ( const regexRuleset of regexRulesets ) {
if ( regexRuleset.enabled !== true ) { continue; }
for ( const rule of regexRuleset.rules ) {
const regex = rule.condition.regexFilter;
const isCaseSensitive = rule.condition.isUrlFilterCaseSensitive === true;
allRules.push(rule);
toCheck.push(dnr.isRegexSupported({ regex, isCaseSensitive }));
}
}
const results = await Promise.all(toCheck);
const newRules = [];
for ( let i = 0; i < allRules.length; i++ ) {
const rule = allRules[i];
const result = results[i];
if ( result instanceof Object && result.isSupported ) {
newRules.push(rule);
} else {
console.info(`${result.reason}: ${rule.condition.regexFilter}`);
}
}
const oldRules = await dnr.getDynamicRules();
const oldRuleMap = new Map(oldRules.map(rule => [ rule.id, rule ]));
const newRuleMap = new Map(newRules.map(rule => [ rule.id, rule ]));
const addRules = [];
const removeRuleIds = [];
for ( const oldRule of oldRules ) {
const newRule = newRuleMap.get(oldRule.id);
if ( newRule === undefined ) {
removeRuleIds.push(oldRule.id);
} else if ( JSON.stringify(oldRule) !== JSON.stringify(newRule) ) {
removeRuleIds.push(oldRule.id);
addRules.push(newRule);
}
}
for ( const newRule of newRuleMap.values() ) {
if ( oldRuleMap.has(newRule.id) ) { continue; }
addRules.push(newRule);
}
if ( addRules.length !== 0 || removeRuleIds.length !== 0 ) {
await dnr.updateDynamicRules({ addRules, removeRuleIds });
}

const dynamicRules = await dnr.getDynamicRules();
console.log(`Dynamic rule count: ${dynamicRules.length}`);

const enabledRulesets = await dnr.getEnabledRulesets();
console.log(`Enabled rulesets: ${enabledRulesets}`);

console.log(`Available dynamic rule count: ${dnr.MAX_NUMBER_OF_DYNAMIC_AND_SESSION_RULES - dynamicRules.length}`);

dnr.getAvailableStaticRuleCount().then(count => {
console.log(`Available static rule count: ${count}`);
});
})();
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@@ -0,0 +1,25 @@
{
"author": "Raymond Hill",
"background": {
"service_worker": "background.js",
"type": "module"
},
"declarative_net_request": {
"rule_resources": [
]
},
"description": "uBO Minus is permission-less experimental MV3-based network request blocker",
"icons": {
"16": "img/icon_16.png",
"32": "img/icon_32.png",
"64": "img/icon_64.png",
"128": "img/icon_128.png"
},
"manifest_version": 3,
"minimum_chrome_version": "101.0",
"name": "uBO Minus (MV3)",
"permissions": [
"declarativeNetRequest"
],
"version": "0.1.0"
}
@@ -0,0 +1,235 @@
/*******************************************************************************
uBlock Origin - a browser extension to block requests.
Copyright (C) 2022-present Raymond Hill
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see {http://www.gnu.org/licenses/}.
Home: https://github.com/gorhill/uBlock
*/

'use strict';

/******************************************************************************/

import fs from 'fs/promises';
import process from 'process';

import rulesetConfigs from './ruleset-config.js';
import { dnrRulesetFromRawLists } from './js/static-dnr-filtering.js';

/******************************************************************************/

const commandLineArgs = (( ) => {
const args = new Map();
let name, value;
for ( const arg of process.argv.slice(2) ) {
const pos = arg.indexOf('=');
if ( pos === -1 ) {
name = arg;
value = '';
} else {
name = arg.slice(0, pos);
value = arg.slice(pos+1);
}
args.set(name, value);
}
return args;
})();

/******************************************************************************/

async function main() {

const writeOps = [];
const ruleResources = [];
const regexRuleResources = [];
const outputDir = commandLineArgs.get('output') || '.';

let goodTotalCount = 0;
let maybeGoodTotalCount = 0;

const output = [];
const log = (text, silent = false) => {
output.push(text);
if ( silent === false ) {
console.log(text);
}
};

const replacer = (k, v) => {
if ( k.startsWith('__') ) { return; }
if ( Array.isArray(v) ) {
return v.sort();
}
if ( v instanceof Object ) {
const sorted = {};
for ( const kk of Object.keys(v).sort() ) {
sorted[kk] = v[kk];
}
return sorted;
}
return v;
};

const isUnsupported = rule =>
rule._error !== undefined;
const isRegex = rule =>
rule.condition !== undefined &&
rule.condition.regexFilter !== undefined;
const isRedirect = rule =>
rule.action !== undefined &&
rule.action.type === 'redirect' &&
rule.action.redirect.extensionPath !== undefined;
const isCsp = rule =>
rule.action !== undefined &&
rule.action.type === 'modifyHeaders';
const isRemoveparam = rule =>
rule.action !== undefined &&
rule.action.type === 'redirect' &&
rule.action.redirect.transform !== undefined;
const isGood = rule =>
isUnsupported(rule) === false &&
isRedirect(rule) === false &&
isCsp(rule) === false &&
isRemoveparam(rule) === false
;

const rulesetDir = `${outputDir}/rulesets`;
const rulesetDirPromise = fs.mkdir(`${rulesetDir}`, { recursive: true });

const fetchList = url => {
return fetch(url)
.then(response => response.text())
.then(text => ({ name: url, text }));
};

const readList = path =>
fs.readFile(path, { encoding: 'utf8' })
.then(text => ({ name: path, text }));

const writeFile = (path, data) =>
rulesetDirPromise.then(( ) =>
fs.writeFile(path, data));

for ( const ruleset of rulesetConfigs ) {
const lists = [];

log(`Listset for '${ruleset.id}':`);

if ( Array.isArray(ruleset.paths) ) {
for ( const path of ruleset.paths ) {
log(`\t${path}`);
lists.push(readList(`assets/${path}`));
}
}
if ( Array.isArray(ruleset.urls) ) {
for ( const url of ruleset.urls ) {
log(`\t${url}`);
lists.push(fetchList(url));
}
}

const rules = await dnrRulesetFromRawLists(lists, {
env: [ 'chromium' ],
});

log(`Ruleset size for '${ruleset.id}': ${rules.length}`);

const good = rules.filter(rule => isGood(rule) && isRegex(rule) === false);
log(`\tGood: ${good.length}`);

const regexes = rules.filter(rule => isGood(rule) && isRegex(rule));
log(`\tMaybe good (regexes): ${regexes.length}`);

const redirects = rules.filter(rule =>
isUnsupported(rule) === false &&
isRedirect(rule)
);
log(`\tredirect-rule= (discarded): ${redirects.length}`);

const headers = rules.filter(rule =>
isUnsupported(rule) === false &&
isCsp(rule)
);
log(`\tcsp= (discarded): ${headers.length}`);

const removeparams = rules.filter(rule =>
isUnsupported(rule) === false &&
isRemoveparam(rule)
);
log(`\tremoveparams= (discarded): ${removeparams.length}`);

const bad = rules.filter(rule =>
isUnsupported(rule)
);
log(`\tUnsupported: ${bad.length}`);
log(
bad.map(rule => rule._error.map(v => `\t\t${v}`)).join('\n'),
true
);

writeOps.push(
writeFile(
`${rulesetDir}/${ruleset.id}.json`,
`${JSON.stringify(good, replacer, 2)}\n`
)
);

regexRuleResources.push({
id: ruleset.id,
enabled: ruleset.enabled,
rules: regexes
});

ruleResources.push({
id: ruleset.id,
enabled: ruleset.enabled,
path: `/rulesets/${ruleset.id}.json`
});

goodTotalCount += good.length;
maybeGoodTotalCount += regexes.length;
}

writeOps.push(
writeFile(
`${rulesetDir}/regexes.js`,
`export default ${JSON.stringify(regexRuleResources, replacer, 2)};\n`
)
);

await Promise.all(writeOps);

log(`Total good rules count: ${goodTotalCount}`);
log(`Total regex rules count: ${maybeGoodTotalCount}`);

// Patch manifest
const manifest = await fs.readFile(`${outputDir}/manifest.json`, { encoding: 'utf8' })
.then(text => JSON.parse(text));
manifest.declarative_net_request = { rule_resources: ruleResources };
const now = new Date();
manifest.version = `0.1.${now.getUTCFullYear() - 2000}.${now.getUTCMonth() * 100 + now.getUTCDate()}`;
await fs.writeFile(
`${outputDir}/manifest.json`,
JSON.stringify(manifest, null, 2) + '\n'
);

// Log results
await fs.writeFile(`${outputDir}/log.txt`, output.join('\n') + '\n');
}

main();

/******************************************************************************/
@@ -0,0 +1,6 @@
{
"engines": {
"node": ">=17.5.0"
},
"type": "module"
}

1 comment on commit a559f5f

@gorhill
Copy link
Owner Author

@gorhill gorhill commented on a559f5f Sep 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The experimental version is now available in the Chrome Web Store (CWS): https://chrome.google.com/webstore/detail/ddkjiahejlhfcafbddmgiahcphecmpfh

The currently published version has no popup panel, but there is a CWS update currently pending review which brings back the ability to disable/enable on a per-site basis.

Please sign in to comment.