Emscripten: mark assets used in the loading map as important

[fdelapena]

Before having an async file implementation, static data preloading was a big file. The problem of this was not just a huge loading time: browser is not caching large files and a page refresh needed to download the whole game data again.

Current async implementation is loading files on demand in the moment they are called. This allows to cache files apart of fast loading but makes the game display/hear the file a bit more later than it should, most noticeable on slow connections, degrading the game user experience.
To prevent this, files should be loaded enough time before by preloading them in some way.

[carstene1ns]

If done correctly, this should use some emscripten magic:
e.g. use FS.createPreloadedFile() for files that are needed before anything else (MapTree, ini... later also Title screen Music/Pictures).
A drawback using this method is, that they have to be known before actual runtime, either when compiling a special version for one game only or when packaging gamedata before bundling.

For the files needed when changing the Map, main loop blockers can be used. However, this needs parsing the entire Map for appropriate files (maybe just the large ones...for example Music?)

This should not affect the async stuff much, as the loading functions may look for files in the virtual filesystem first, before actually downloading (emscripten_*wget*) anything.

[Ghabry]

Are you still unhappy about the async loading strategy or can that be closed?

[fdelapena]

This one could be deferred IMO. As carstene1ns pointed, critical files could be preloaded, e.g. for "The Sacred Tears: TRUE" picture based cutscene or the "+13" animation in "Desolate City".

[Ghabry]

The only direct improvement I see for map changes is to declare all files directly required when the map loads as important so that the transition looks better (without stuff popping in).

[fdelapena]

Another hacky but simple approach to achieve faster asset loading without messing with Player code:

• Generate via JavaScript a list of <link rel=prefetch> on load, by reading the JSON generated with gencache.

Most browsers have prefetching enabled by default, except those not wanting to prefetch content (e.g. for privacy reasons or bandwidth savings).

[elsemieni]

• Generate via JavaScript a list of <link rel=prefetch> on load, by reading the JSON generated with gencache.

Sounds very reasonable, I want to add a few things to consider about.

• This is kinda subjective but can help in end-user perception of load time: There's some categories of resources that are "more important" than others when a game loads. Let's say, a player will sense the game is loading faster by load several sorts of resources first (database, system, chipset, charas, sounds used in database, etc), then load the least important ones (pictures, animations), leaving at the end the most expensive and optional resources (again, the priority is subjective) like music. The con of this is giving priorities can be complex, as in different projects several sort of resources are more important than others.

• It can also be good idea considering start loading resources (given the JSON cache already) before/meanwhile player binary loads (I don't know if that makes sense, as maybe some code of the player binary are needed first). Not waiting for player's bootup can let us some extra time to load resources meanwhile. I mean, this loading code can be a separate JS code in the player index.

[fdelapena]

For the first point, a simplest and fine tuned approach would be editing the generated json file by hand if needed, instead of adding sophisticated priority heuristics.

About preloading JSON assets before Player WASM binary, makes sense, it could start loading before the large binary to make the asset loading experience smoother, defining a javascript code (inline or external script loaded from the header) the index.html.

[fdelapena]

Firefox 85 released on January 26th, introducing full <link rel="preload"> support. Because prefetch does not guarantee the download, preload makes it more interesting. It is already supported by Blink (Chrome, Edge) and WebKit (Safari), so it looks as a the simpler workaround to improve the web player experience.

[Ghabry]

Have some simple idea to solve it: The Web Player could look for a "preload.zip" file on startup in the game directory. This zip is extracted into the game directory and everything that was in there is obviously not downlaoded again.

Problem: The game author is responsible for this, but when you look at "nico" they are already doing this anyway to make there game web-compatible.

And our streaming solution works for 99% of the RPG Maker games already. Only problem are "fast" scenes. Do not want to cause lots of extra bandwidth by downloading everything on a map, even stuff that is never visible. Waste of traffic.

ZIP reading is already available in the ZIP VFS. Extracting could be done by our ZIP handler. (Want to avoid something like a "OverlayFS", another source of errors)

[jetrotal]

Hi, the yno project uses a collection of preload JSON files to deal with files that must be loaded in order to not break animations or so:
https://github.com/ynoproject/ynopreloads

[Ghabry]

afaik the preloader does not preload anymore since the last Player update xD