TST bloats Firefox session store file

[tobbez]

Short description

The (uncompressed) size of Firefox session store file (sessionstore.jsonlz4) gets significantly bloated when using Tree Style Tabs.

This is mainly due to these keys that it adds, that are stored in extData in the session file:

extension:treestyletab@piro.sakura.ne.jp:cached-sidebar-contents
extension:treestyletab@piro.sakura.ne.jp:cached-tabs

For example, after decompressing the lz4 compression, my session file was 257MiB. After removing these keys:

extension:treestyletab@piro.sakura.ne.jp:cached-sidebar-contents
extension:treestyletab@piro.sakura.ne.jp:cached-sidebar-contents:tabs-dirty
extension:treestyletab@piro.sakura.ne.jp:cached-sidebar-contents:collapsed-dirty
extension:treestyletab@piro.sakura.ne.jp:cached-tabs

it shrunk to just 38MiB.

Why this is a problem

Once a session file becomes too large, it appears Firefox refuses to load it (even in safe mode), making it appear as if all the tabs have been lost.

Firefox failed to load the 257MiB file described in the previous section, but after stripping the TST cache related keys, it loaded correctly.

Admittedly, this is with a massive amounts of tabs, and you could argue that it's a bug in Firefox. However, without TST, you would need something like 20k+ tabs to make the session file grow to a similar size.

In addition to the above, I'm guessing that Firefox also has problems writing session files that are too large, because after crashing, the data in the recovery file was about one and a half month old (though it had been written just before the crash).

Environment

• Platform (OS): Windows
• Version of Firefox: 61.0b8 (was the same on 59.0b6 as well)
• Version (or revision) of Tree Style Tab: 2.4.21

See also

Bug filed for Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1464685

Addendum

In case someone else who had the same issue finds this, here's what I did to make the session store loadable by Firefox again:

First use https://github.com/andikleen/lz4json to decompress the session file:

git clone https://github.com/andikleen/lz4json
cd lz4json
make
./lz4jsoncat sessionstore.jsonlz4 > sessionstore_decompressed.js

Then use jq (https://stedolan.github.io/jq/) to remove the cache-related keys:

jq 'del(.windows[].tabs[].extData["extension:treestyletab@piro.sakura.ne.jp:cached-sidebar-contents"]) | del(.windows[].tabs[].extData["extension:treestyletab@piro.sakura.ne.jp:cached-sidebar-contents:tabs-dirty"]) | del(.windows[].tabs[].extData["extension:treestyletab@piro.sakura.ne.jp:cached-sidebar-contents:collapsed-dirty"]) | del(.windows[].tabs[].extData["extension:treestyletab@piro.sakura.ne.jp:cached-tabs"]) | del(.windows[]._closedTabs[].state.extData["extension:treestyletab@piro.sakura.ne.jp:cached-sidebar-contents"]) | del(.windows[]._closedTabs[].state.extData["extension:treestyletab@piro.sakura.ne.jp:cached-sidebar-contents:tabs-dirty"]) | del(.windows[]._closedTabs[].state.extData["extension:treestyletab@piro.sakura.ne.jp:cached-sidebar-contents:collapsed-dirty"]) | del(.windows[]._closedTabs[].state.extData["extension:treestyletab@piro.sakura.ne.jp:cached-tabs"])' sessionstore_decompressed.js > sessionstore.js

Then place sessionstore.js in the Firefox profile directory. It should be possible to use either sessionstore.js, sessionstore.jsonlz4 or any file in sessionstore-backups in the Firefox profile directory as a base.

[SXZ1]

I currently have 11 000 tabs open, sessionstore.lz4json is just 5 mb in size. I guess that's because I disabled caching in TST settings.

[piroor]

Due to some reasons, now TST stores caches of the window to the last tab. So, if you open new last tab and close it again and again, stored tab data for the "undo close tab" command will be increased...

[polyzen]

@piroor, you should consider using the help wanted label instead:
https://help.github.com/articles/about-labels/#using-default-labels

[ngirard]

I've regularly experienced Firefox refusing to load my session file, until I spent several hours today nailing down the problem to this very issue.

@tobbez's detailed explanations were a life saver: thanks very much !

After stripping TST's cache metadata from my session file, Firefox loaded it flawlessly.

Also the size of my recovery.jsonlz4 file went from 42 Mb down to 9.5 Mb.

I'm also suspecting that this bloating of my session data was the reason why Firefox's memory footprint was periodically increasing and decreasing when backing up the session file, triggering the Linux kernel's out-of-memory killer on a daily basis.

I cannot but hope that a solution is found to this problem, making these bitter experiences a thing of the past.

Meanwhile, I'm disabling caching in TST settings. Could we have an explanation about the consequences of disabling caching ?

In any case, I wish to take the opportunity to thank @piroor once again for his continuous efforts in maintaining TST. Cheers !

[piroor]

I've introduced some changes around the cache system. TST now tries to clear cache data of non-last tabs if possible. Due to WebExtensions's restriction, TST cannot clear cache data of already removed tabs notified via tabs.onRemoved event. However, if you close tabs via TST's UI (context menu, closebox in tabs, and middle-click on tabs) TST tries to clear cache before closing them.

[ngirard]

Thanks for these changes !

Will the cache be cleared when using Ctrl-w to close tabs ?

[piroor]

@ngirard Sadly no, Ctrl-W case is not handled for now. Ctrl-W is handled by Firefox itself, and Firefox just reports an event indicating the tab is removed, and TST cannot clear cache data for the tab. (Even if TST tries do that, WebExtensions API just returns an error like "the tab is not exist".)

[ngirard]

Thanks for your explanations. It looks like a tricky situation...

[irvinm]

@piroor is there anything more that can be done at this point or is this a "bug of Firefox itself" for now?

[piroor]

@irvinm To be honest I forgot why I gave up to store cache data per window instead of per tab, so I've reintroduce changes to store caches as an window value. You can activate this change with enabling a secret option storeCacheAsWindowValue to true. It should reduce the size of stored data, but there may be some regressions. After some dogfooding I think I should make a decision to activate or deactivate it by default.

[irvinm]

Has anyone who has had problems with their session store file getting large try and comment on the new option? My session store file is only 4.3MB with 1003 tabs so I will not be trying.

[irvinm]

@piroor I believe this issue could be considered "expired" ...

[piroor]

I close this because there looks to be no progress.

[bb010g]

I've had this issue occur recently, with large cached-tabs entries inside the extData of multiple tabs, along with cached-sidebar-contents over 88_641_471 bytes long decoded & uncompressed. (One of the big issues with cached-sidebar-contents is duplication of large favicon Base64 data strings between <tab-item data-current-favicon-url="data:…">, child <tab-favicon src="data:…">, and <tab-favicon> child <img src="data:…">.)

I have a somewhat redacted sessionstore.json file from a real-world session of 2_574 tabs, 334_165_589 bytes uncompressed & 87_592_311 bytes compressed, that I can send to @piroor if they'd be willing to take it. (For the curious, see my redaction jq filter at the end. I don't have the time to completely, manually redact that much data, and I don't want to mess up anything valuable for performance analysis.) It'd also be better for analysis of TST-caused slowdown than what I was able to provide for #2278 in the past. TST would sometimes take minutes to finish loading the sidebar & show tabs, with high CPU usage during loading.

For now, I'm going to try running TST with the cache totally disabled on a basically empty session that I manually cleaned of all TST extData.

redact-fxsession.jq

# Helpers

def log(x): .;
# def log(x): . as $dot | x | debug | $dot;

def modify_paths(paths; update): reduce paths as $p (.; setpath($p; getpath($p) | update));
def modify(path_expressions; update): modify_paths(path(path_expressions); update);

def index_if(condition; key; default): if condition then .[key] else default end;
def index_if(condition; key): index_if(condition; key; empty);
def index_if_has($key; default): index_if(has($key); $key; default);
def index_if_has($key): index_if(has($key); $key);

# def if_filter(cond; f): if cond then f else . end;
# def with_selected_entries(boolean_expression; f):
#   to_entries | if boolean_expression then 
#   f | 
#   from_entries;

# Extension redaction

"treestyletab@piro.sakura.ne.jp" as $ext_treeStyleTab |
"alltabshelper@alltabshelper.org" as $ext_allTabsHelper |

def redact_tab_treeStyleTab: log("redact_tab_treeStyleTab") |
  .;

def redact_tab_allTabsHelper: log("redact_tab_allTabsHelper") |
  .;

def redact_tab_extData: log("redact_tab_extData") |
  redact_tab_treeStyleTab |
  redact_tab_allTabsHelper |
  .;

def redact_win_treeStyleTab: log("redact_win_treeStyleTab") |
  .;

def redact_win_allTabsHelper: log("redact_win_allTabsHelper") |
  .;

def redact_win_extData: log("redact_win_extData") |
  redact_win_treeStyleTab |
  redact_win_allTabsHelper |
  .;

# Firefox redaction

def redact_entry: log("redact_entry") |
  del(.presState) |
  del(.structuredCloneState) |
  modify(index_if_has("children"); modify(.[]; redact_entry));

def redact_tab: log("redact_tab") |
  modify(index_if_has("entries"); modify(.[]; redact_entry)) |
  modify(index_if_has("extData"); modify(.[]; fromjson) | redact_tab_extData | modify(.[]; tojson)) |
  del(.formdata) |
  del(.scroll) |
  del(.storage) |
  del(.userTypedClear) |
  del(.userTypedValue) |
  .;

def redact_win: log("redact_win") |
  modify(index_if_has("extData"); modify(.[]; fromjson) | redact_win_extData | modify(.[]; tojson)) |
  del(.height) |
  del(.screenX) |
  del(.screenY) |
  modify(index_if_has("tabs", "_closedTabs"); modify(.[];
    redact_tab |
    modify(index_if_has("state"); redact_tab) |
  .)) |
  del(.width) |
  .;

modify(index_if_has("windows", "_closedWindows"); modify(.[]; redact_win)) |
del(.cookies)

[ngirard]

Thanks for your efforts, @bb010g. Hopefully they'll allow to make further progress. Keep us updated !

[tuukka]

I just had this same issue with TST version 3.8.16.

For me, the bulk of the data was in _closedTabs which I removed with

z4jsoncat sessionstore.jsonlz4 | jq -c 'del(..|._closedTabs?)' >sessionstore.js

This took the uncompressed size from 281M to 51M.