New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support gzip for html games #394

Open
leafo opened this Issue Apr 15, 2016 · 13 comments

Comments

Projects
None yet
6 participants
@leafo
Member

leafo commented Apr 15, 2016

more information here: https://cloud.google.com/storage/docs/json_api/v1/how-tos/performance

We need to have gzip ahead of time since google cloud storage can't zip on the fly. I think it makes most sense to have zipserver do the work. Another option is to push everything through cloudflare

@goldfire

This comment has been minimized.

goldfire commented Apr 15, 2016

+1 for CloudFlare. We've used them in production for several years with no issues. We also recently did a benchmark and CloudFlare was head of the pack among CDNs (which is especially great since it is free).

@eska014

This comment has been minimized.

eska014 commented Apr 3, 2017

Any news on this? Godot Engine will support WebAssembly coming next version, the binary compresses very well, shrinks from ~12MB down to ~3MB

@fasterthanlime

This comment has been minimized.

Member

fasterthanlime commented Apr 3, 2017

Btw since we last posted here, CloudFlare is no longer an option (for internal reasons).

I'm sure our actual CDN has an equivalent though.

@leafo

This comment has been minimized.

Member

leafo commented Apr 3, 2017

We'll have to update our zip extractor to upload the gz files and preserve the mime type. Most engines already export as pre-zipped package. Cloudflare isn't an option here since we use more traffic than what is allowed by their flat fee plans.

@eska014 does the Godot HTML export automatically generate a compressed file? Or is it depending on the server to do it?

@eska014

This comment has been minimized.

eska014 commented Apr 3, 2017

@leafo Godot currently generates several uncompressed files (index.html, engine blob, game content, some JS) into a selected directory.
Users are expected to zip up the files for uploading to itch.io, the server is expected to take care of compression.

We can implement an export-time toggle to generate one zip archive with gzipped files, would uploading that to an itch.io page just work? Not sure how compression works with a CDN 'in between'

@fasterthanlime

This comment has been minimized.

Member

fasterthanlime commented Apr 3, 2017

@eska014 we can configure the CDN to gzip-compress some file extensions. What extension does the engine blob have?

@leafo

This comment has been minimized.

Member

leafo commented Apr 3, 2017

The way unity works is that it compresses things ahead of time, creating jsgz or memgz files. In our case we aren't detecting their content type correctly, and it's set to just octet stream. Unity actually has a gzip extractor built into their html5 runtime so they extract it after downloading within the browser. I'm going to update our zip extractor to set the correct content type header, that was the original purpose of this issue.

http://leafo.net/shotsnb/2017-04-03_14-25-36.png

In the future, I think we should just automatically gzip all files we upload to our storage that are used for HTML5 games.

As for your approach, @eska014, I think doing something like what Unity does is probably the best approach. Your average user is either going to skip comprssion, or add something on their webserver that compresses during the request, which will puts unnecessary pressure on the server. Ahead of time compression seems like a good way to make sure they're serving compressed files.

@eska014

This comment has been minimized.

eska014 commented Apr 4, 2017

@fasterthanlime The filename extension for the engine is .asm.js for the current asm.js-based export plus a separate .mem file for static memory. The upcoming WebAssembly format uses .wasm.
The game content uses either .pck for Godot's custom binary format, or a zip archive with either a .zip or .pcz extension.

@leafo
In the future, I think we should just automatically gzip all files we upload to our storage that are used for HTML5 games.

If that's possible, that would be great

@Beuc

This comment has been minimized.

Beuc commented Jun 3, 2018

I tested a bit using emscripten and web assembly.
I mainly need to serve index.js, index.wasm and index.data.
On my test server I pre-compress them so I have index.js.gz, index.wasm.gz, index.data.gz.

  1. Ideally callling xxx.ext would automatically look for pre-compressed xxx.ext.gz, leaving the original "Content-type", plus adding "Content-Encoding: gzip" so the browser transparently decodes it. This is not currently the case at itch.io's CDN, you get a 403 error.

  2. Alternatively, you'd fiddle with emscripten's output and the program would directly call the .gz, adding a "Content-Encoding: gzip" so the browser can decompress it. This is not currently the case at ioch.io's CDN, the browser just tries to interpret the gzip'd stream without decompressing, so this fails.

  3. Incidentally, for wasm to be efficiently loaded, it needs the "Content-type: application/wasm". itch.io's CDN returns application/octet-stream so we get an error "wasm streaming compile failed: TypeError: Response has unsupported MIME type" (which can be ignored though, just not efficient).

Currently the only way that works is upload all js/wasm/data uncompressed AND serve them uncompressed to the user. I probably could embed a JS-based gunzip but that sounds like troubles for my .data file(s) (~120M uncompressed with 50% gzip ratio), not to mention I'd have to patch emscripten's initialization process.

I can get all 3 points with a 4-line .htaccess, though I understand configuring a CDN is a more involved process (but I'm surprised this hasn't been tackled before!).

Would it be possible to implement at least point 2 ("Content-Encoding: gzip")?
Otherwise what do you recommend?

@fasterthanlime

This comment has been minimized.

Member

fasterthanlime commented Jun 3, 2018

(but I'm surprised this hasn't been tackled before!

I remember @leafo and I spending a good amount of time on that, trying various combinations, and eventually noticed it broke all Unity WebGL exports (with some now-deprecated but already-widely-used-for-export version of Unity. They do ship a javascript inflater as a fallback, btw).

My personal take is that:

  • HTTP contains all we need to serve gzip resources
    • Rewriting xxx.ext to xxx.ext.gz feels wrong (we shouldn't do path mapping)
  • Transparently serving a gzip'd (or brotli'd) version of assets is the CDN's job
  • Transparently decompressing an asset is the browser's job

The problem is that some game engines have encouraged people to upload already-compressed
versions of assets to their web servers, and add non-trivial configuration, so we're forced to find a solution for that scenario too, which is probably not going to be an option, but something based on automatic detection.


The situation right now (doing nothing special) works for recent Unity WebGL games, ie.:

  • Unity WebGL builds already-gzipped asset files
  • They're served directly, without Content-Encoding: gzip
  • Unity's runtime falls back to its own javascript decompression
    • Which is a waste of CPU cycles but hey, bandwidth is saved.

This is just a drive-by update, as both leaf & I are deep into other issues at the moment, but I just wanted to reassure you that it's on our mind. Everything is, always :shipit:

@Beuc

This comment has been minimized.

Beuc commented Jun 3, 2018

Thanks for the update!
I suspected some backward compatibility issue indeed. Btw xxx.ext->www.ext.gz is not rewriting but content negotiation/MultiViews really.

@leafo

This comment has been minimized.

Member

leafo commented Jun 4, 2018

our CDN is supposed to be transparently gzip compressing files for whitelisted extensions, .js is one of them. (It's not applied to everything since it was breaking unity games). But I'm checking now and not seeing it. Going to debug...

@GSGBen

This comment has been minimized.

GSGBen commented Sep 16, 2018

Hi, any update on specifying gzip encoding in the headers? UE4 outputs really small builds when using it but unfortunately can't run on itch:

Downloading failed: 
Downloaded a compressed file GT.data.jsgz without the necessary HTTP response header "Content-Encoding: gzip" specified!
Please configure gzip compression on this asset on the web server to serve gzipped assets!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment