Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: "The extension failed to load properly. It might not be able to intercept network requests." #1653

Open
Tracked by #1669
philipp-classen opened this issue Jun 11, 2024 · 36 comments
Assignees
Labels
Bug Something isn't working Needs investigation

Comments

@philipp-classen
Copy link
Member

philipp-classen commented Jun 11, 2024

Update from Ghostery:

The Google team is aware of this problem and provided a fix in Chromium 124. Please ensure that you are running the latest version of Chrome, or any other browser that is based on Chromium.

For Windows 7/8/8.1 users: The last version of Chromium is 109, which is causing known problems. Since Windows 7/8/8.1 is end-of-life and will no longer receive any updates, it is strongly recommended to upgrade to Windows 10/11. If that is not possible, a temporary solution is to switch to Firefox ESR, which still provided updates.


After rolling out Ghostery 10 (migrating to Manifest V3), we got reports on the Chrome store about this error message:

The extension failed to load properly. It might not be able to intercept network requests.

Unfortunately, we cannot reproduce yet, nor did the reports provide additional details. This issue is intended as a placeholder and to collect additional information.

Firefox should not be affected (still on Ghostery 8). It has been reported on the Chrome store, but it is likely that it affects other Chromium-forks.

@philipp-classen philipp-classen added the Bug Something isn't working label Jun 11, 2024
@philipp-classen philipp-classen pinned this issue Jun 11, 2024
@ghostwords
Copy link

It's probably chrome.storage.local.get() not returning an object, and/or chrome.storage.local being undefined on browser startup. More Chromium bugs caused by basing extensions on service workers.

https://groups.google.com/a/chromium.org/g/chromium-extensions/c/cbERJLSL11A

@philipp-classen philipp-classen self-assigned this Jun 11, 2024
@philipp-classen
Copy link
Member Author

@ghostwords Thanks for the link! I'm not sure what happen in Ghostery if chrome.storage.local being temporarily unavailable at service worker startup. We don't have monitoring to detect such errors at the moment, so I can neither confirm nor rule it out.

Another possibility that we are investigating: if multiple extensions are installed, perhaps the global rule limit could be exceeded and causing this error.

@philipp-classen
Copy link
Member Author

I cannot confirm that we are affected; but as an experiment, I added this before the startup:

chrome.storage.local = undefined;

As expected, the extension no longer works, but the error messages are slightly different:

TypeError: Cannot read properties of undefined (reading 'set')
Error while observing options: TypeError: Cannot read properties of undefined (reading 'set')
Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'set')
Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'set')
Error in event handler: TypeError: Cannot read properties of undefined (reading 'pipelineSteps') at RequestReporter.recordClick (chrome-extension://pgmhiegmkbenpafgojfccjnbhponejmg/node_modules/@whotracksme/webextension-packages/packages/reporting/src/request-reporter.js:95:25) at chrome-extension://pgmhiegmkbenpafgojfccjnbhponejmg/background/reporting/webrequest-reporter.js:81:26
Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'get')
TypeError: Cannot read properties of undefined (reading 'get')

From a user perspective, you can still use the UI and it appears as if it is correctly set up; but the adblocker does not work. So, it may be a different problem, but could explain some of generic reports stating that nothing is working.

@philipp-classen
Copy link
Member Author

philipp-classen commented Jun 13, 2024

Could be coincidence, but so far the reports that included details were all for Chrome on Windows 7. The sample size is very small, but it could be the core of the problem.

Note: Chrome is no longer supported on Windows 7 (source: https://support.google.com/chrome/a/answer/7100626?hl=en):

Chrome 109 is the last version to support this operating system.
Chrome 109 was released January 10, 2023.

@CDR-DPH
Copy link

CDR-DPH commented Jun 14, 2024

Windows 10 patched thru end of May 2024.

Error Msg: The extension failed to load properly. It might not be able to intercept network requests.

Error in event handler: TypeError: Cannot read properties of undefined (reading 'pipelineSteps') at RequestReporter.recordClick (chrome-extension://mlomiejdfkolichcflejclcbmpeaniij/node_modules/@whotracksme/webextension-packages/packages/reporting/src/request-reporter.js:95:25) at chrome-extension://mlomiejdfkolichcflejclcbmpeaniij/background/reporting/webrequest-reporter.js:85:26
Context
Unknown
Stack Trace
:0 (anonymous function)


DNR - Error while updating lists: Error: Internal error.
Context
background/dnr.js
Stack Trace
background/dnr.js:63 (anonymous function)

import { observe, isGlobalPaused, ENGINES } from '../store/options.js';
import { TRACKERDB_ENGINE } from '../utils/engines.js';

/**

  • Ghostery Browser Extension
  • https://www.ghostery.com/
  • Copyright 2017-present Ghostery GmbH. All rights reserved.
  • This Source Code Form is subject to the terms of the Mozilla Public
  • License, v. 2.0. If a copy of the MPL was not distributed with this
  • file, You can obtain one at http://mozilla.org/MPL/2.0
    */

const PAUSE_RULE_PRIORITY = 10000000;

{
const DNR_RESOURCES = chrome.runtime
.getManifest()
.declarative_net_request.rule_resources.filter(({ enabled }) => !enabled)
.map(({ id }) => id);

// Ensure that DNR rulesets are equal to those from options.
// eg. when web extension updates, the rulesets are reset
// to the value from the manifest.
observe(null, async (options) => {
const globalPause = isGlobalPaused(options);
const ids = ENGINES.map(({ name, key }) => {
return !globalPause && options.terms && options[key] ? name : '';
}).filter((id) => id && DNR_RESOURCES.includes(id));

if (ids.length) {
  ids.push(TRACKERDB_ENGINE, 'fixes');
}

const enabledRulesetIds =
  (await chrome.declarativeNetRequest.getEnabledRulesets()) || [];

const enableRulesetIds = [];
const disableRulesetIds = [];

for (const id of ids) {
  if (!enabledRulesetIds.includes(id)) {
    enableRulesetIds.push(id);
  }
}

for (const id of enabledRulesetIds) {
  if (!ids.includes(id)) {
    disableRulesetIds.push(id);
  }
}

if (enableRulesetIds.length || disableRulesetIds.length) {
  try {
    await chrome.declarativeNetRequest.updateEnabledRulesets({
      enableRulesetIds,
      disableRulesetIds,
    });
    console.info('DNR - Lists successfully updated');
  } catch (e) {
    console.error(`DNR - Error while updating lists:`, e);
  }
}

});

observe('paused', async (paused, prevPaused) => {
// Skip if hostnames has not changed

@philipp-classen
Copy link
Member Author

philipp-classen commented Jun 14, 2024

@CDR-DPH Thanks for the report!

Can you please share, which Browser and which version you are using?

DNR - Error while updating lists: Error: Internal error.

I have seen this error message. It looks like the browser bug described here: #1554

I thought it is fixed and will automatically recover, so I closed it. But perhaps I was too early. Since I was affected by it as well, I can confirm that adblocking is then completely broken. Uninstalling and reininstalling the extension may help.


Looking at the bug reports, we had a single other report on Windows 10, but lots of reports for Windows 7/8. So, it is safe to say that there is a problem on Windows >=10; but given that practically all reports are from 7/8 - now with only a few percent market share - it looks like there are independent problems.

@CDR-DPH
Copy link

CDR-DPH commented Jun 14, 2024 via email

@philipp-classen
Copy link
Member Author

Using SRWare’s Iron bowerer – chromium derivative. Version 122.0.0 I have reinstalled ghostery numerous times but no luck.

I see there is a version 125.0.0.0 listed on their website. Would be interesting if it makes a difference.

Some background: the problem with DNR - Error while updating lists: Error: Internal error. was first reported on Chromium 120 in Oct 2023 (https://issues.chromium.org/issues/40285683). A fix came end of March, which I believe should be included in Chromium 124.

@Printninja
Copy link

Printninja commented Jun 15, 2024

Can confirm this error appears on Chrome Version 109.0.5414.120 running on Windows 7. Removed and reinstalled the extension multiple times. Still get the error.

@lakrsrool
Copy link

lakrsrool commented Jun 17, 2024

I have the same exact problem with Ghostery on the Vivaldi browser [6.5.3206.39 (Stable channel) (64-bit)]. Reinstalling extension does not help.

Is the CSS code posted previously (above) supposed to "fix" this issue for now?

@lakrsrool
Copy link

lakrsrool commented Jun 17, 2024

^ Followup - (to reiterate, reinstalling add-on does not help) - beyond this, if I disable/enable the add-on the error will disappear momentarily but soon (w/in seconds) return.

I have tried disabling "Ad Blocking" option - no help (same results).

@philipp-classen
Copy link
Member Author

I have the same exact problem with Ghostery on the Vivaldi browser [6.5.3206.39 (Stable channel) (64-bit)]. Reinstalling extension does not help.

Is the CSS code posted previously (above) supposed to "fix" this issue for now?

@lakrsrool Note that Vivaldi is based on Chromium. Could you try to update Vivaldi to the latest version? The current version that I see on Windows and Linux is 6.7.3329.41.

For the 6.7 version, I see Chrome/124.0.0.0 being shown as part of the user agent. The bugfix came in Chromium 124. I assume 124.0.0.0 is a truncated version number and Vivaldi 6.7.3329.41 should have the fix, but I'm not familiar enough with Vivaldi to confirm that.

Since it is a browser bug, unfortunately we cannot do anything in the Ghostery extension to work around it. You have to upgrade to a Chromium version that includes the fix.

@Printninja
Copy link

So basically there's no way to use the latest version of Ghostery on older versions of Chrome (i.e. Windows 7) since Chrome cannot be updated on Windows 7 past version 109.0.5414.120

@philipp-classen
Copy link
Member Author

@Printninja Yes, both Windows 7 and also 8.1.

In 2023, Microsoft ended their support and Google followed shortly after for Chrome. Since there are no more security updates, our strong recommendation is for everyone to upgrade to Windows 10/11. Or to consider switching to Linux (especially an option for older hardware).

@Printninja
Copy link

I am aware that Windows 7 is no longer supported by Microsoft, but I prefer the Windows 7 O.S. for a number of reasons. I do not like Windows 10/11. Microsoft deliberately exaggerates the "security" issue to try and force people into using Windows 10.

@lakrsrool
Copy link

lakrsrool commented Jun 18, 2024

I have the same exact problem with Ghostery on the Vivaldi browser [6.5.3206.39 (Stable channel) (64-bit)]. Reinstalling extension does not help.
Is the CSS code posted previously (above) supposed to "fix" this issue for now?

@lakrsrool Note that Vivaldi is based on Chromium. Could you try to update Vivaldi to the latest version? The current version that I see on Windows and Linux is 6.7.3329.41.

For the 6.7 version, I see Chrome/124.0.0.0 being shown as part of the user agent. The bugfix came in Chromium 124. I assume 124.0.0.0 is a truncated version number and Vivaldi 6.7.3329.41 should have the fix, but I'm not familiar enough with Vivaldi to confirm that.

Since it is a browser bug, unfortunately we cannot do anything in the Ghostery extension to work around it. You have to upgrade to a Chromium version that includes the fix.

Thanks for the feedback.

I've updated Vivaldi to 6.7.3329.41 - and the error for the Ghostery extension has disappeared. So it appears this update has addressed the issue.

Many thanks again.

I'm thinking about going back to auto-updates (I had just recently changed to my getting a notice - but I don't recall ever getting a notice that an update was available).

@i300220
Copy link

i300220 commented Jun 18, 2024

You could reproduce anytime with Slimjet 42.0.3.0. Your last 3 versions of ghostery triggered this error. I tried to upgrade to slimjet 43.0.3.0 but it's so badly made that even though I had no more ghostery extension error, the computer was unusable, so back to 42.0.3.0 on Fedora 40.

And I'm in disaccord as it is a browser bug. I've been running Slimjet 42.0.3.0 since Feb 1 2024 and only the version 10 of ghostery caused issues. Issues started on June 12 precisely then stopped the following day and are back again as of today the 18th.

Version 42.0.3.0 (basé sur Chromium 120.0.6099.18) (Build officiel) (64 bits)

I'm extremely reluctant to ever upgrade slimjet when it's not broken.

@i300220
Copy link

i300220 commented Jun 21, 2024

Ghostery 10.3.9 error details: Cannot load extension with file or directory name _metadata. Filenames starting with "_" are reserved for use by the system.

@chax-chax
Copy link

how do i roll back on older ghostery? (that still works on chrome 109.0.5414.168)
been using this for a long time, i forgot how annoying ads were. until this happens.

i'd rather not install another extension.

@Printninja
Copy link

how do i roll back on older ghostery? (that still works on chrome 109.0.5414.168) been using this for a long time, i forgot how annoying ads were. until this happens.

i'd rather not install another extension.

You can download older version from their github page and install them manually into the older version of Chrome. I did it successfully, but it doesn't seem to stay installed. I'm not sure why, but the extension seems to disappear shortly after I install it.

I gave up and just installed uBlock Origin.

@chax-chax
Copy link

I gave up and just installed uBlock Origin.

i guess this is the only way?
welp, thanks ghostery, it's been a good 10 years i think? pretty sure i started using before 2015.

@Printninja
Copy link

Or you go to Windows 10 (ugh) and then you can use the latest Chrome browser. I've already built a Windows 10 tower. Just need to do a data transfer and set everything up. Lots of work. I hate the idea of abandoning my trusty Windows 7 machine, but more and more software simply won't run under Windows 7. Frustrating.

Also, there's always Firefox.

@i300220
Copy link

i300220 commented Jun 21, 2024

i guess this is the only way?

You can download ghostery v8, put your browser extensions into developer mode, it's important. Then unzip to a custom location where it doesn't risk to be deleted, outside of your browser dirs.

Then on the extension page, click on 'load unpacked extension', point it to the path where you unzipped it, and that's it. It will stay installed until you choose to update it manually.

@i300220
Copy link

i300220 commented Jun 21, 2024

more and more software simply won't run under Windows 7. Frustrating.

Time to get linux a chance. I to, run either linux or Win 7, dual boot. Won't ever upgrade Win. Haven't booted Win in months now...

@Printninja
Copy link

Won't ever upgrade Win. Haven't booted Win in months now

You can't run Adobe Creative Cloud software under Linux or Windows 7.

I own a graphic design and web development company, and as much as I hate it, I have to be able to use the latest versions of Adobe software when clients send me files. If Linux were a practical option, I'd have switched in a heartbeat long ago.

@i300220
Copy link

i300220 commented Jun 21, 2024

Unfortunately true for Adobe Software. Sad!

Curiosity if you want to give it a try: I have a Win 7 laptop and ghostery 10.3.6 works perfectly fine with Slimjet Version 38.0.10.0 (based on Chromium 109.0.5414.74) (Official Build) (32-bit).

@chax-chax
Copy link

You can download ghostery v8 ...
... until you choose to update it manually.

this works, thank you so much :)

@philipp-classen
Copy link
Member Author

Ghostery 10.3.9 error details: Cannot load extension with file or directory name _metadata. Filenames starting with "_" are reserved for use by the system.

@i300220 I also noticed that message recently. So, it is not lost, I split out this issue: #1687

@philipp-classen
Copy link
Member Author

philipp-classen commented Jun 21, 2024

You could reproduce anytime with Slimjet 42.0.3.0. Your last 3 versions of ghostery triggered this error. I tried to upgrade to slimjet 43.0.3.0 but it's so badly made that even though I had no more ghostery extension error, the computer was unusable, so back to 42.0.3.0 on Fedora 40.

And I'm in disaccord as it is a browser bug. I've been running Slimjet 42.0.3.0 since Feb 1 2024 and only the version 10 of ghostery caused issues. Issues started on June 12 precisely then stopped the following day and are back again as of today the 18th.

Version 42.0.3.0 (basé sur Chromium 120.0.6099.18) (Build officiel) (64 bits)

I'm extremely reluctant to ever upgrade slimjet when it's not broken.

@i300220 This the bug that I was referring to: https://issues.chromium.org/issues/40285683

It has been fixed in later Chromium versions (124 or higher), but it still exists in 120 (or earlier). It worked with Ghostery 8 because it is using Manifest V2, but since 3rd of June Google Chrome forces all extensions to upgrade to Manifest V3. When using the new DNR API, this bugfix is critical.

@i300220
Copy link

i300220 commented Jun 21, 2024

This the bug that I was referring to: https://issues.chromium.org/issues/40285683

I see, thanks. Slimjet, contrarily to Brave or Vivaldi is not updated very frequently. And the last version
is catastrophic.

But strangely enough, Ghostery 10.3.6 works perfectly fine on Win 7 with Slimjet Version 38.0.10.0 (based on Chromium 109.0.5414.74) (Official Build) (32-bit).

@sparky672
Copy link

sparky672 commented Jun 22, 2024

Ghostery 10.3.9
Mac High Sierra (10.13.6)
Chrome Version 114.0.5735.198 (Official Build) (x86_64)

I can confirm this situation as well. Some days, clicking "reload" will fix it; but other days, it does not matter how many times I hit "reload".

I've reinstalled - no change.

I've also disabled all other extensions so Ghostery is the only one - again, no change.

This is super frustrating and not a good experience.

EDIT: Using Developer Mode, I rolled it back to 10.3.6 and it seems to be working.

@kshep
Copy link

kshep commented Jun 23, 2024

Just noticed this 10.3.9 in Chrome Version 126.0.6478.115 (Official Build) (arm64) on macOC Sonoma Version 14.5 on an M3 MacBook Air.

I did a "Load unpacked" of 10.3.8 and it seems to work fine except there's an error...

Refused to load the font 'data:font/truetype;charset=utf-8;base64,... because it violates the following Content Security Policy directive: "font-src github.githubassets.com".

at content_scripts/autoconsent.js:957 (hideElements)

@i300220
Copy link

i300220 commented Jun 23, 2024

4 new Ghostery 10.3.9 errors today:

Slimjet Version 42.0.3.0 (basé sur Chromium 120.0.6099.18) (Build officiel) (64 bits)

Failed to update engine "ads" TypeError: Cannot read properties of undefined (reading 'lists')
Contexte
utils/engines.js
Trace de la pile
utils/engines.js:361 (fonction anonyme)
import { openDB } from '../node_modules/idb/build/index.js';
import FilterEngine, { ENGINE_VERSION } from '../node_modules/@cliqz/adblocker/dist/esm/src/engine/engine.js';
import '../node_modules/@cliqz/adblocker/dist/esm/src/data-view.js';
import '../node_modules/@cliqz/adblocker/dist/esm/src/request.js';
import '../node_modules/@cliqz/adblocker/dist/esm/src/filters/cosmetic.js';
import '../node_modules/@cliqz/adblocker/dist/esm/src/filters/network.js';
import { getLinesWithFilters, mergeDiffs } from '../node_modules/@cliqz/adblocker/dist/esm/src/lists.js';
import '../node_modules/@cliqz/adblocker/dist/esm/src/fetch.js';
import Config from '../node_modules/@cliqz/adblocker/dist/esm/src/config.js';
import '../node_modules/@remusao/small/dist/es6/index.js';
import { observe } from '../store/options.js';
import { registerDatabase } from './indexeddb.js';
import debug from './debug.js';

/**
 * Ghostery Browser Extension
 * https://www.ghostery.com/
 *
 * Copyright 2017-present Ghostery GmbH. All rights reserved.
 *
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0
 */


const CUSTOM_ENGINE = 'custom-filters';
const FIXES_ENGINE = 'fixes';
const TRACKERDB_ENGINE = 'trackerdb';

const engines = new Map();

const ENV = new Map([
  ['ext_ghostery', true],
  ['cap_html_filtering', checkUserAgent('Firefox')],
  ['env_firefox', checkUserAgent('Firefox')],
  ['env_chromium', checkUserAgent('Chrome')],
  ['env_edge', checkUserAgent('Edg')],
  ['env_mobile', checkUserAgent('Mobile')],
  ['env_experimental', false],
]);

observe('experimentalFilters', (experimentalFilters) => {
  ENV.set('env_experimental', experimentalFilters);

  // As engines on the server might have new filters, we need to update them
  updateAll().catch(() => null);

  for (const engine of engines.values()) {
    engine.updateEnv(ENV);
  }
});

function checkUserAgent(pattern) {
  return navigator.userAgent.indexOf(pattern) !== -1;
}

function deserializeEngine(engineBytes) {
  const engine = FilterEngine.deserialize(engineBytes);
  engine.updateEnv(ENV);
  return engine;
}

function parseFilters(filters) {
  const config = new Config({
    enableHtmlFiltering: ENV.get('cap_html_filtering'),
  });
  const engine = FilterEngine.parse(filters, config);
  engine.updateEnv(ENV);
  return engine;
}

function loadFromMemory(name) {
  return engines.get(name);
}

function saveToMemory(name, engine) {
  engines.set(name, engine);
}

// custom filter exceptions should apply to all engines
function shareExceptions(name, engine) {
  if (name === CUSTOM_ENGINE || name === FIXES_ENGINE) return;

  // Network exceptions
  const matchExceptions = engine.exceptions.match.bind(engine.exceptions);
  engine.exceptions.match = (...args) => {
    return (
      get(CUSTOM_ENGINE).exceptions.match(...args) ||
      get(FIXES_ENGINE).exceptions.match(...args) ||
      matchExceptions(...args)
    );
  };

  // Cosmetic exceptions
  const iterMatchingFiltersUnhide =
    engine.cosmetics.unhideIndex.iterMatchingFilters.bind(
      engine.cosmetics.unhideIndex,
    );
  engine.cosmetics.unhideIndex.iterMatchingFilters = (...args) => {
    iterMatchingFiltersUnhide(...args);
    get(FIXES_ENGINE).cosmetics.unhideIndex.iterMatchingFilters(...args);
    get(CUSTOM_ENGINE).cosmetics.unhideIndex.iterMatchingFilters(...args);
  };

  const matchAllUnhideExceptions = engine.hideExceptions.matchAll.bind(
    engine.hideExceptions,
  );
  engine.hideExceptions.matchAll = (...args) => {
    return (
      matchAllUnhideExceptions(...args) ||
      get(FIXES_ENGINE).hideExceptions.matchAll(...args) ||
      get(CUSTOM_ENGINE).hideExceptions.matchAll(...args)
    );
  };
}

const DB_NAME = registerDatabase('engines');

async function getDB() {
  if (!getDB.current) {
    getDB.current = openDB(DB_NAME, 1, {
      upgrade(db) {
        db.createObjectStore('engines');
      },
      async blocking() {
        const db = await getDB.current;
        db.close();
        getDB.current = null;
      },
    });
  }

  return getDB.current;
}

async function loadFromStorage(name) {
  try {
    const db = await getDB();

    const tx = db.transaction('engines');
    const table = tx.objectStore('engines');

    const engineBytes = await table.get(name);

    if (engineBytes) {
      const engine = deserializeEngine(engineBytes);
      shareExceptions(name, engine);
      saveToMemory(name, engine);

      return engine;
    }
  } catch (e) {
    console.error(`Failed to load engine "${name}" from storage`, e);
  }

  return null;
}

async function saveToStorage(name) {
  try {
    const engine = loadFromMemory(name);
    const db = await getDB();

    const tx = db.transaction('engines', 'readwrite');
    const table = tx.objectStore('engines');

    await table.put(engine.serialize(), name);
  } catch (e) {
    console.error(`Failed to save engine "${name}" to storage`, e);
  }
}

function check(response) {
  if (!response.ok) {
    throw new Error(
      `Failed to fetch engine "${name}": ${response.status}: ${response.statusText}`,
    );
  }

  return response;
}

const CDN_HOSTNAME = chrome.runtime.getManifest().debug
  ? 'staging-cdn.ghostery.com'
  : 'cdn.ghostery.com';

async function update(name) {
  if (name === CUSTOM_ENGINE) return;

  try {
    const urlName =
      name === 'trackerdb'
        ? 'trackerdbMv3'
        : `dnr${"chrome" === 'firefox' ? '' : '-cosmetics'}-${name}`;

    const data = await fetch(
      `https://${CDN_HOSTNAME}/adblocker/configs/${urlName}/allowed-lists.json`,
    )
      .then(check)
      .then((res) => res.json());

    if (!data.engines[ENGINE_VERSION]) {
      throw new Error(
        `Engine "${name}" for "${ENGINE_VERSION}" engine version not found`,
      );
    }

    // Get current engine
    let engine = loadFromMemory(name);

    // Check if some lists need to be removed from the engine: either because
    // there are lists removed from allowed-lists.json or because some region
    // lists need to be disabled. In this case, we just reset the engine for
    // simplicity. Doing so also allows us to save memory because we do not have
    // to keep track of which filters belong to which list.
    //
    // We also make sure that all lists which need to be updated have an
    // available diff. If not, the operation would be equivalent to first
    // deleting the list then adding the new version. Because of this, we also
    // reset the engine if that happens.
    let foundListsToRemove = false;
    for (const [name, checksum] of engine.lists.entries()) {
      // If engine has a list which is not "enabled"
      if (!data.lists[name]) {
        foundListsToRemove = true;
        break;
      }

      // If engine has an out-dated list which does not have a diff available
      if (
        data.lists[name].checksum !== checksum &&
        data.lists[name].diffs[checksum] === undefined
      ) {
        foundListsToRemove = true;
        break;
      }
    }

    // Make a full update if we need to remove some lists
    if (foundListsToRemove) {
      const arrayBuffer = await fetch(data.engines[ENGINE_VERSION].url)
        .then(check)
        .then((res) => res.arrayBuffer());

      const engineBytes = new Uint8Array(arrayBuffer);
      engine = deserializeEngine(engineBytes);
      shareExceptions(name, engine);
      // Save the new engine to memory and storage
      saveToMemory(name, engine);
      saveToStorage(name);

      console.log(`Engine "${name}" reloaded`);

      return engine;
    }

    // At this point we know that no list needs to be removed anymore. What
    // remains to be done is: *add new lists* and *update existing lists with
    // their respective diffs*.
    const diffs = [];

    /**
     * Helper function used to fetch a full list, parse it, accumulate
     * parsed filters, then update the checksum in engine if previous
     * steps were successful.
     */
    const fetchListToAdd = async ({ name, checksum, url }) => {
      try {
        // Create new diff and update version of the list in `this.engine`
        diffs.push({
          added: Array.from(
            getLinesWithFilters(
              await fetch(url)
                .then(check)
                .then((res) => res.text()),
              engine.config,
            ),
          ),
        });
        engine.lists.set(name, checksum);
      } catch (e) {
        console.error(`Failed to add list "${name}"`, e);
      }
    };

    /**
     * Helper function used to fetch a list diff, parse it, accumulate
     * parsed filters, then update the checksum in engine if previous
     * steps were successful.
     */
    const fetchListToUpdate = async ({ name, checksum, url }) => {
      try {
        // Create new diff and update version of the list in engine
        diffs.push(
          await fetch(url)
            .then(check)
            .then((res) => res.json()),
        );
        engine.lists.set(name, checksum);
      } catch (e) {
        console.error(`Failed to update list "${name}"`, e);
      }
    };

    // Go over enabled list and start fetching the ones which need to be added
    // or updated. All of this will happen concurrently.
    const promises = [];
    for (const name of Object.keys(data.lists)) {
      const checksum = engine.lists.get(name);
      if (checksum === undefined) {
        promises.push(
          fetchListToAdd({
            name,
            checksum: data.lists[name].checksum,
            url: data.lists[name].url,
          }),
        );
      } else if (checksum !== data.lists[name].checksum) {
        promises.push(
          fetchListToUpdate({
            name,
            checksum: data.lists[name].checksum,
            url: data.lists[name].diffs[checksum],
          }),
        );
      }
    }

    // Wait for all lists to have been fetched and parsed
    await Promise.all(promises);

    // `engine.update` method will return `true` if anything was
    // updated and `false` otherwise.
    const cumulativeDiff = mergeDiffs(diffs);
    let updated = engine.updateFromDiff(cumulativeDiff, ENV);

    // Last but not least, check if resources.txt should be updated. This can be
    // done independently of filters as the data is stored in a separate object.
    if (
      data.resources &&
      data.resources.checksum !== engine.resources.checksum
    ) {
      engine.updateResources(
        await fetch(data.resources.url)
          .then(check)
          .then((res) => res.text()),
        data.resources.checksum,
      );
      updated = true;
    }

    if (updated) {
      console.log(`Engine "${name}" updated`);
      // Save the new engine to storage
      saveToStorage(name);
    }

    return engine;
  } catch (e) {
    console.error(`Failed to update engine "${name}"`, e);
    throw e;
  }
}

function updateAll() {
  return Promise.all(Array.from(engines.keys()).map((name) => update(name)));
}

async function loadFromDisk(name) {
  try {
    const response = await fetch(
      chrome.runtime.getURL(
        `rule_resources/engine-${name}${
          "chrome" === 'firefox' || name === 'trackerdb' ? '' : '-cosmetics'
        }.dat`,
      ),
    );

    const engineBytes = new Uint8Array(await response.arrayBuffer());
    const engine = deserializeEngine(engineBytes);
    shareExceptions(name, engine);
    saveToMemory(name, engine);
    saveToStorage(name);

    // After initial load from disk, schedule an update
    // as it is done only once on the first run.
    // After loading from disk, it should be loaded from the storage
    update(name).catch(() => null);

    return engine;
  } catch (e) {
    console.error(`Failed to load engine "${name}" from disk`, e);
    return new FilterEngine();
  }
}

function get(name) {
  return loadFromMemory(name);
}

const ALARM_PREFIX = 'engines:update:';
const ALARM_DELAY = 60; // 1 hour

async function init(name) {

  if (name === CUSTOM_ENGINE) {
    return (
      get(CUSTOM_ENGINE) ||
      (await loadFromStorage(CUSTOM_ENGINE)) ||
      (await createCustomEngine())
    );
  }

  // Schedule an alarm to update engines once a day
  chrome.alarms.get(`${ALARM_PREFIX}${name}`, (alarm) => {
    if (!alarm) {
      chrome.alarms.create(`${ALARM_PREFIX}${name}`, {
        delayInMinutes: ALARM_DELAY,
      });
    }
  });

  return (
    get(name) || (await loadFromStorage(name)) || (await loadFromDisk(name))
  );
}

async function createCustomEngine(filters = '') {
  const engine = parseFilters(filters);

  saveToMemory(CUSTOM_ENGINE, engine);
  saveToStorage(CUSTOM_ENGINE);

  return engine;
}

chrome.alarms.onAlarm.addListener((alarm) => {
  if (alarm.name.startsWith(ALARM_PREFIX)) {
    const name = alarm.name.slice(ALARM_PREFIX.length);
    update(name).catch(() => null);

    chrome.alarms.create(alarm.name, {
      delayInMinutes: ALARM_DELAY,
    });
  }
});

debug.engines = { get };

export { CUSTOM_ENGINE, FIXES_ENGINE, TRACKERDB_ENGINE, createCustomEngine, get, init, updateAll };
 
Error in event handler: TypeError: this.tabs.get(...).setActive is not a function at onTabActivated (chrome-extension://mlomiejdfkolichcflejclcbmpeaniij/node_modules/@whotracksme/webextension-packages/packages/reporting/src/webrequest-pipeline/page-store.js:124:28)

Error in event handler: TypeError: Cannot read properties of undefined (reading 'pipelineSteps') at RequestReporter.recordClick (chrome-extension://mlomiejdfkolichcflejclcbmpeaniij/node_modules/@whotracksme/webextension-packages/packages/reporting/src/request-reporter.js:95:25) at chrome-extension://mlomiejdfkolichcflejclcbmpeaniij/background/reporting/webrequest-reporter.js:86:26

Error in event handler: TypeError: tabContext.setActive is not a function at onTabUpdated (chrome-extension://mlomiejdfkolichcflejclcbmpeaniij/node_modules/@whotracksme/webextension-packages/packages/reporting/src/webrequest-pipeline/page-store.js:96:16)

@Piotr-23
Copy link

May you please fix this error in Chrome for Windows 7... Using it on a modern Mac at work and an old Win 7 at home causes problems - if the extension is removed from one device, it disappears on the second....

@chrmod
Copy link
Member

chrmod commented Jun 28, 2024

@Piotr-23 sadly there is no way we can support Ghostery for older versions of Chrome. For now it would be best if you disable Extension synchronization from your Chrome profile. You should be able to find it in chrome://settings/?search=sync
We do strongly recommend you update your browser, this is a matter of general security. If there is no way to install latest Chrome on Window 7, then the best choice would be to switch to something like Firefox ESR

@i300220
Copy link

i300220 commented Jul 8, 2024

Interestingly, the Brave browser will support MV2 extensions for another year thanks to the 'ExtensionManifestV2Availability’ key.

https://github.com/brave/brave-core/blob/master/chromium_src/extensions/common/extension_features.cc

https://brave.com/blog/brave-shields-manifest-v3/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Needs investigation
Projects
None yet
Development

No branches or pull requests