Compress HTML5 export for hosts that don't support on-the-fly compression like itch.io

[paulloz]

Hey there.

It would be nice if the Html5 export had a option to automatically gzip both the .wasm and the .pck output files. Currently you can do that afterwards but you'll need to hack into the exported .js file to ungzip the .wasm so it is not optimal (ungzipping the .pck is probably possible from the Html shell).

Usually you'd rely on your server to gzip those files but for instance https://itch.io/ (which is really popular for distributing web games) does not do it because other engines (e.g. Unity) already compress their data during the export.

[akien-mga]

CC @eska014

[leonkrause]

I don't think this should be an option, it should always be enabled or it'll just cause confusion. A web frontend fallback must be added for webgame hosts that deliver without Accept-Encoding headers.

We should also consider Brotli, Calinou tested this and it seems there is a reasonable difference in size.

Not sure if compressing the main pack makes sense, doesn't .pck already use some compression?

[vnen]

Not sure if compressing the main pack makes sense, doesn't .pck already use some compression?

AFAIK it doesn't have any compression.

[LinuxUserGD]

But it is already possible to provide the game pack as .zip which is compressed.

[Ajver]

This works fine for me: https://www.reddit.com/r/godot/comments/8b67lb/guide_how_to_compress_wasmpck_file_to_make_html5/

[paulloz]

Hi!
I'm still interested in this issue (and in implementing a solution) but I'd like some pointers, maybe from core contributors. Afaik most (if not all?) solutions will require to embark a compression tool, whether it is brotli, pako or any other implementation. I'm not sure about implications in terms of licenses, how it should be included in the codebase etc.

[Calinou]

Brotli's license makes it fine for inclusion in Godot. Integrating Brotli would also make it possible to support WOFF2 fonts as discussed in #38588.

That said, we already have Zstandard in Godot, so we might want to reuse it instead of adding yet another compression library.

[paulloz]

@Calinou thanks! To be fair, I my point was more about the frontend side and what will be used to uncompress the files.
IIRC we also have Gzip available in addition to Zstandard too.

[Zireael07]

This turns out to be more important that we thought, as Godot HTML5 games load slowly and every few seconds that can be shaved off would massively improve player experience.

Whichever compression library ends up used, it ought to help.

[Calinou]

@Zireael07 I'm not sure if compressing the WASM payload and letting Godot decompress it would be any faster than letting the Web server send a compressed version and have the browser decompress it on-the-fly. It might be faster, but it's a lot of work to get it working in the first place.

Paging @kripken as he might know the tradeoffs here.

[kripken]

Usually you'd rely on your server to gzip those files but for instance https://itch.io/ (which is really popular for distributing web games) does not do it because other engines (e.g. Unity) already compress their data during the export.

This makes me sad... I think some engines started to do that because not all hosting providers enabled serverside compression. If major websites like itch.io are not doing it because engines already do then that's a really bad feedback loop. The engine doing it will never be as fast as the browser + webserver working together in the standard way 😢

If you do want to do this in Godot, then if the pck file is a normal file from emscripten's point of view, you can compress it using the LZ4 option, with just -s LZ4 (but obviously LZ4 is not the most compact compression format - the decompressor is tiny though, and very fast).

For the wasm file, you'd need something external to emscripten. That may have different tradeoffs, though, as if you compress it manually you may be preventing the browser from doing streaming compilation (compile during the download). So that might be worth measuring - a bigger download might start up faster in some cases if it's streaming. There are a bunch of decompressors in JS that could be experimented with (like pako and zee).

[paulloz]

If you're interested in this, there's a WIP branch on my fork repo with a partial implementation.

[NHodgesVFX]

This would be good to have for itch.io or any server you dont control that doesnt have server side compression enabled. Although it should be an option.

I did some comparisons between compression algorithms

All compression algorithms used the maximum settings, only the .pak and the .wasm were compressed. the pak was always 35kb. Files were compressed using https://peazip.github.io/

Custom export template with most everything striped the .wasm was 9.75mb. Release build

7z lzma2 - 1.8mb
Zstd - 2mb
brotili - 2.11mb
gzip - 2.55mb

with the default export template, release build. the size of the .wasm is 13.4mb

7z lzma2 - 2.65mb
Zstd - 2.94mb
brotili - 3.12mb
gzip - 3.75mb

Based on these results I think adding compression to html5 export would be good. its seems the best would be zstd as it maintains good compression and is built into godot.

[paulloz]

Sorry, my last reply was quickly made from my phone and it seems that branch wasn't pushed on my remote but it is now.
Basically it lacks client-side decompression to be good to go. For now it'd mean embarking a decompression .js lib to do this but I'm not sure if there's a better way to handle it. Compression method can be selected in the export window (none, zstd or gzip).

[NHodgesVFX]

I dont know enough about JS to add it but here is the library for zstd.

https://www.npmjs.com/package/zstd-codec

it it says its unpacked size is 3mb im assuming that can be minimized to make it smaller.

[fuckgithubanyway]

Has anything changed in three years?

[Calinou]

Has anything changed in three years?

No, although we have support for Brotli decompression in Godot now. That said, using Zstandard from Godot would still be a better choice if we want to support compressed PCKs that are decompressed at load-time (Zstandard is more efficient, faster, or both).

The .wasm file itself requires a different approach (decompression from JavaScript) as Godot isn't initialized at the time the browser reads that file.

[fuckgithubanyway]

Brotli seems to be a less effective (maybe even completely irrelevant) tool for this task. Can you suggest some simple way to decompress the .wasm file of the engine itself in the body of the main html file of the games (or so), which does not require deep knowledge in the field of rocket science and quantum physics, for simple workers like me?

[Calinou]

Can you suggest some simple way to decompress the .wasm file of the engine itself in the body of the main html file of the games (or so), which does not require deep knowledge in the field of rocket science and quantum physics, for simple workers like me?

Look into using pako as mentioned above, but this won't be trivial. @paulloz's branch no longer appears to be available so this is something you'd have to implement yourself.

[paulloz]

Yeah, sorry. I cleaned my repo a few weeks ago and considered that was too old. I think a dumb version could be quite easy to get working. Unless there are major roadblocks in the Godot 4 export process (I haven't looked at it at all, I have no idea how different it is from Godot 3).

[miltoncandelero]

Recently I wrote a rant blog post on how unity does webgl compression, why it is doing it wrong and how to do it right and it was brought to my attention that Godot could benefit from it.

The post in question:

https://miltoncandelero.github.io/unity-webgl-compression

Where it was brought to my attention

JohannesDeml/UnityWebGL-LoadingTest#151

I will look more into how the Godot build process work and what files are actually generated and report back

[Calinou]

Note that web browsers have started adding Zstandard support since the end of last year. Safari is currently missing support for it though.