Investigate compressing omni.ja with LZMA2

[the-r3dacted]

I'm probably going to ship Brotli compressed omni.ja files next release, which decreases the installed size by a substantial 50MB for 115 ESR, but this increases the installer/.7z size a little bit. (4.4MB for 115 ESR)

The installer and the .7z file are compressed with LZMA2 which causes this discrepancy. However LZMA2 generally gives better compression and is more universal so that's why it's used for those.

Which means if I were to also use LZMA2 for the omni.ja files, this would cause a negligible installer/.7z file size increase (0.2MB for 115 ESR) and be even smaller than the Brotli compressed counterpart installed on disk. (A further 5MB or 55MB smaller than no compression for 115 ESR. This actually is exactly a 1/4th size reduction for the browser (219MB extracted))

There looks to be some capability of using LZMA2 in Firefox, the installer uses it, and xz is included in the source code. However I'm not sure how to use it here.

[the-r3dacted]

@e3kskoy7wqk I wonder is this something you'd be interested in?

[e3kskoy7wqk]

Before you raised this issue, I had never paid attention to the size of omni.ja. I didn't even know what omni.ja was for.

However, if it can decrease the installed size by a substantial 50MB for 115 ESR, that would be truly amazing.

[Vangelis66]

... Brotli compession in omni.ja files has been used in recent years in official Pale Moon by MCP (to "protect" their copyrighted branding sources) and I, personally, as an advanced user, am against it, because the standard decompressing tools (e.g. 7-zip)/scripts can't extract the br-compressed ".ja"s for further user customisation; this is one of the reasons, by popular demand, that the roytam1 UXP forks have kept the originally used zip compression for their omni.jas 😉 ...

Mozilla now use by default Store compression, but Deflate:Maximum can also be used to reduce filesize to ca. 34% of the original: zip -oqr9XD omni.ja *

[the-r3dacted]

I was going to suggest 7-Zip ZS, but apparently that has errors extracting omni.ja that I didn't notice before. Ig I'll report an issue on their GitHub.

[the-r3dacted]

I get the feeling the errors might be caused by the --lgwin=17 in mozjar.py

set LZ77 window size (0, 10-24)
window size = 2**NUM - 16
0 lets compressor choose the optimal value
use incompatible large-window brotli
bitstream with window size (0, 10-30)
WARNING: this format is not compatible
with brotli RFC 7932 and may not be
decodable with regular brotli decoders

[the-r3dacted]

Nope, still get errors with no lgwin parameter.

It's actually slightly smaller without it, but the compressed .7z is 8MB bigger, so ig I will keep --lgwin

I also decided to test with 0 since it says it will choose the optimal value, and I can confirm that 0 is slightly smaller than 17, so ig I will go with that.

[the-r3dacted]

Huh lgwin=0 is the same size as no lgwin extracted, but it's significantly smaller than no lgwin in .7z form.

[Vangelis66]

I'm probably going to ship Brotli compressed omni.ja files next release

Brotli compession in omni.ja files has been used in recent years in official Pale Moon by MCP (to "protect" their copyrighted branding sources) and I, personally, as an advanced user, am against it, because the standard decompressing tools (e.g. 7-zip)/scripts can't extract the br-compressed ".ja"s for further user customisation

I was going to suggest 7-Zip ZS, but apparently that has errors extracting omni.ja

Posting it here for future, eventual, reference 😉 :

To the best of my knowledge, the only tool (actually, a Python script) able to successfully extract brotli-compressed omni.ja browser asset files is the one below:

https://github.com/JustOff/mozjarr

AIUI, the Mozilla-authored code applies special optimisation routines during br-compression (e.g. it moves some special index headers to the beginning of the file, so that the browser engine doesn't need to fully decompress the file at runtime to get info about its contents) and this applied optimisation renders the archive non-spec compliant and, as a consequence, incompatible with standard extraction tools...

The releases section offers PyInstaller-compiled Windows binaries; mozjarr(32|64).exe are built with py3.8, so they need Win7SP1+; mozjarr64_legacy.exe is built with py3.7, so it needs VistaSP2_x64+; for VistaSP2_x86, one has to use mozjarrXP.exe, built on py2.7_x86...

Background (OT)

JustOff, a Ukrainian, was a member of the MCP team, also a regular contributor to the Waterfox Classic project; let's just say that his "relationship" with Moonchild (and then important MCP contributor Matt. A. Tobin) had its ups and downs, but eventually he was unceremoniously expelled from the MCP team (then responsible for Pale Moon, Basilisk and the UXP platform) over major disagreement on the direction(s) development should follow...

mozjarr was developed after JustOff's "exit", possibly to "spite" 😉 Moonchild over his decision to "encrypt" PM's omni.jas (using Brotli compression) ...

JustOff's "fate" is unknown since the start of the Russia-Ukraine conflict; for all I know, he might've died in the war 😿 ...