Settings ignored as added lists reloaded after Firefox launch in private mode

[mraandtux]

Prerequisites

• I verified that this is not a filter list issue. Report any issues with filter lists or broken website functionality in the uAssets issue tracker.
• This is not a support issue or a question. For support, questions, or help, visit /r/uBlockOrigin.
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue.
• The issue is not present after disabling uBO in the browser.
• I checked the documentation to understand that the issue I am reporting is not normal behavior.

I tried to reproduce the issue when...

• uBO is the only extension.
• uBO uses default lists and settings.
• using a new, unmodified browser profile.

Description

Mind the false on the "suspendUntilListsAreLoaded" and "autoUpdate" settings. The "added" lists ignores the global setting ("auto update" and "uspend network activity until all filter lists are loaded" were unticked) and reloading from the web after launch.

uBlock Origin: 1.50.1b12
Firefox: 115
...
listset (total-discarded, last-updated):
 removed:
  urlhaus-1: null
  plowe-0: null
  CHN-0: null
 added:
  https://easylist-downloads.adblockplus.org/easylistchina.txt: 22775-89, never
  fanboy-cookiemonster: 47808-228, never
 default:
  user-filters: 149-0, never
  ublock-filters: 34627-55, never
  ublock-badware: 7482-0, never
  ublock-privacy: 464-0, never
  ublock-unbreak: 2102-0, never
  ublock-quick-fixes: 270-5, never
  easylist: 67761-309, never
  easyprivacy: 33394-87, never
filterset (user): [array of 34 redacted]
userSettings:
 autoUpdate: false
 cloudStorageEnabled: true
 cnameUncloakEnabled: false
 showIconBadge: false
 suspendUntilListsAreLoaded: false
hiddenSettings: [none]
supportStats:
 allReadyAfter: 6783 ms
 maxAssetCacheWait: 324 ms
 unsuspendAfter: 1077 ms

A specific URL where the issue occurs.

N/A

Steps to Reproduce

1. Open Firefox in private mode (firefox --private)
2. Add lists outside of the default lists (UBlock, Easylist, Easyprivacy, etc...) , eg locals and annoyances list
3. restart

Expected behavior

Non-default lists like CHN-0 and fanboy-cookiemonster will loaded from cache

Actual behavior

The lists keeps updating everytime I launch my browser in private mode, bypassing the extension's global settings.

uBO version

1.50.1b12

Browser name and version

Firefox 115 (both PPA deb and Flatpak)

Operating System and version

Ubuntu 23.04

[gorhill]

Unable to reproduce.

The lists keeps updating everytime I launch my browser in private mode

Those symptoms is what you would get if uBO was not able to save locally to indexedDB storage:

• Lists always out of date at each restart
• Imported lists have to be fetched at each restart
• Both points above will lead to longer initialization time since some lists will have to be fetched from remote servers, and all lists will have to be re-compiled

Since I can't reproduce, you need to investigate on your side why your version of uBO is not able to persist data to its indexedDB. See what the Firefox console reports.

[AnteBucan]

I have the same problem. If you always start Firefox in private mode, lists are always out of date and initialization time is longer.

Xubuntu 23.04
Firefox 115 flatpak
Firefox 115 snap
uBlock Origin 1.50.0

If you disable always starting in private mode and use clear everything on exit option, uBlock Origin works fine.

[AnteBucan]

I installed Firefox 115 on Windows 10 in a virtual machine, configured Firefox to always start in private mode and installed uBlock Origin 1.50.0 from Firefox Addons site. The same thing happens. Every time you start Firefox, filters are out of date and uBlock Origin updates them.

[gorhill]

Best is that whoever can reproduce investigate why uBO can't write to its local indexedDB -- just pointing out that you can reproduce won't advance resolving the issue.

[AnteBucan]

Sorry about that. You couldn't reproduce the problem and I had a flatpak and a snap version of Firefox that sometimes behave in strange way because of the sandboxing, so I did a clean install on Windows 10 and reproduced the problem.

I investigated a little bit more and the problem is in Firefox 115. When I install Firefox 114.0.2 and disable automatic updates, everything is working as expected. When i upgrade to Firefox 115, uBlock Origin stops working as expected if you enable "Always use private browsing mode" or use "--private" startup switch (you need to close all the normal windows first).

In Firefox 115 they added:

IndexedDB is now also supported in private browsing without memory limits thanks to encrypted storage on disk. The temporary keys to decrypt the information are held in RAM only and all stored information is purged at the normal end of a private browsing session from disk.

Maybe when they added that functionality they broke something else. I don't know if you can fix this on your end.

[gorhill]

I was still on Firefox 114 in my distro when I tested, and following your comment, I checked and saw there was a pending update for Firefox 115, and now I can reproduce. If what you report is the issue, that would be a regression in Firefox since I am pretty sure the indexedDB of extensions is not supposed to be targeted.

Normally uBO would fall back to using browser.storage.local for its cache storage when it detects that indexedDB is not available for use, but currently there are no error thrown as a result of using indexedDB because it is available for use, it's just that it is removed once the browser quits.

However there is a manual workaround: to
set cacheStorageAPI to browser.local.storage in Advanced settings.

I will have to think about if/how to address this unexpected behavior in the code.

[gorhill]

Related bugzilla issues:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1841806
• https://bugzilla.mozilla.org/show_bug.cgi?id=1639542

[gorhill]

I added a new heuristic in 1.50.1b15, which works as follow:

If all the three following conditions are true:

• Browser is Firefox
• Context is private mode
• indexedDB has not yet been created

Then uBO will use browser.storage.local API to cache data instead of indexedDB API.

I would like to have feedback about whether this works well.

[AnteBucan]

Fresh Firefox 115 installation with uBlock Origin 1.50.1b15 on Windows 10.

Firefox icon is copied and "--private" startup switch is added to it. Firefox is started with normal icon and changed icon.

NORMAL - normal Firefox icon
PRIVATE - Firefox icon with "--private" startup switch
ALWAYS PRIVATE - Firefox with "Always use private browsing mode" (in this case icon with switch "--private" doesn't matter. It's the same as normal icon)

• NORMAL - (first run) filters are out of date
• NORMAL - filters are OK
• PRIVATE - filters are out of date
• NORMAL - filters are OK
• PRIVATE - filters are out of date
• PRIVATE - filters are OK
• NORMAL - filters are OK
• PRIVATE - filters are out of date
• NORMAL - filters are OK
• ALWAYS PRIVATE - filters are out of date
• ALWAYS PRIVATE - filters are OK
• ALWAYS PRIVATE - filters are OK
• NORMAL - filters are OK
• PRIVATE - filters are out of date

(If the filters are out of date, I manually update the filters for speed. The same thing would happen if I waited for automatic update)

When you start Firefox in normal mode, filters are always OK. If you start in private mode after normal mode, filters are always out of date. If you start private mode two times in a row, filters are OK.

If I'm understanding this correctly, normal mode always uses indexedDB, and it persists across changing of modes. Private mode uses browser.storage.local and it gets erased when you change to normal.

Could you use indexedDB for normal mode and browser.storage.local for private mode without erasing it. The storage would be in two places, but everything would be working as expected. If Firefox fixes deletion of extensions indexedDB you could change to only using indexedDB.

[mountaround]

@gorhill

I would like to have feedback about whether this works well

very gladly, with beta 1.50.1b15 the issue is no longer present. (Browser is Firefox, Context is private mode, indexedDB has not yet been created)

System / Settings:
ArchLinux (the arch way)
Firefox 115.0.1

Thank you very much for your very good work, gorhill.

[gorhill]

@AnteBucan Thanks for the thorough testing. From your results, I gather that when Firefox is started in private mode, Firefox uses a separate indexedDB than the one used in normal mode, and consequently the issue is still present when alternating between private and normal mode.

I don't think keeping two cache storages is going to work though, the code assumes there is only one cache storage present and this could lead to seriously broken uBO because uBO keeps a version identifiers of currently cached compiled lists and memory snapshot in its normal storage, and having an unexpectedly stale cache storage could result in a mismatch between the version identifiers recorded in its normal storage and the data loaded from the stale cache storage.

[DitherMan420]

Yep I'm having the same exact issue as everyone in Firefox. My guess is a recent firefox update broke functionality.

@gorhill will this be fixed in a future version of ublock origin? Or does mozilla have to fix this on their end?

[iam-py-test]

While this does need to be fixed by Mozilla, gorhill has put out a temporary fix which I believe is currently in beta.
You need to install the latest beta: https://github.com/gorhill/uBlock/releases/tag/1.50.1b17

[Remus76]

Update from Mozilla:

• Fixed and verified with FF 117 (also latest beta/nightly)
• Fixed and verified with FF 115.2esr (older never affected)