Load engine and data concurrently #8682

ekharkunov · 2024-03-18T15:07:33Z

Now dmloader.js loads game data and game engine (wasm + js files) concurrently.

The default template builtins/manifests/web/engine_template.html has been changed. If you use your own custom template, make sure to update it with these changes. Pay attention to css changes for progress bar and how progress bar animation now handling. Also change how extra_params passed according to new documentation https://defold.com/manuals/html5/#extra-parameters

Fixes #7456.
Possible fixes #7039.

Technical changes

Rework html5 loading flow. Now dmloader.js loads archive_list.json, then loads data and engine concurrently.
Additionally part of js code was moved from engine_template.html to dmloader.js.
dmloader.js used as template during bundle process. It means that some of project values can be used to substitute inside template (see dmloader.js for more details).
Add animation transition for progress bar width change to update progress bar more smooth. Made with transform property (thanks to @thinknathan ).

Manual update: #8682

PR checklist

Example of a well written PR description:

Start with the user facing changes. This will end up in the release notes.
Add one of the GitHub approved closing keywords
Optionally also add the technical changes made. This is information that might help the reviewer. It will not show up in the release notes. Technical changes are identified by a line starting with one of these:
1. ### Technical changes
2. Technical changes:
3. Technical notes:

There was a anomaly in the carbon chroniton propeller, introduced in version 8.10.2. This fix will make sure to reset the phaser collector on application startup.

### Technical changes
* Pay special attention to line 23 of phaser_collector.clj as it contains some interesting optimizations
* The propeller code was not taking into account a negative phase.

ekharkunov · 2024-03-18T15:07:52Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/Project.java

@@ -999,14 +999,7 @@ public void buildEnginePlatform(IProgress monitor, File buildDir, File cacheDir,

        final String variant = appmanifestOptions.get("baseVariant");

-        List<String> defaultNames = platform.formatBinaryName("dmengine");
-        List<File> exes = new ArrayList<File>();


ekharkunov · 2024-03-18T15:09:24Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/HTML5Bundler.java

@@ -203,7 +215,7 @@ void writeJson(JsonGenerator generator) throws IOException {

            generator.writeFieldName("pieces");
            generator.writeStartArray();
-            int offset = 0;


Fix type because file size returns long.

ekharkunov · 2024-03-18T15:09:36Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/HTML5Bundler.java

@@ -257,7 +272,6 @@ public void bundleApplication(Project project, Platform platform, File bundleDir
        String enginePrefix = BundleHelper.projectNameToBinaryName(title);

        BundleHelper.throwIfCanceled(canceled);
-        File projectRoot = new File(project.getRootDirectory());


ekharkunov · 2024-03-18T15:10:36Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/HTML5Bundler.java

            generator.writeEndArray();
+            generator.writeNumberField("total_size", totalSize);


Calculate total game data size to avoid unnecessary calculation during runtime.

ekharkunov · 2024-03-18T15:11:35Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

-                xhr.onprogress = function(e) {
-                    if (onprogress) onprogress(xhr, e);
+                xhr.onprogress = function(event) {
+                    if (onprogress) onprogress(xhr, event, xhr._loadedSize);


Pass already loaded amount of bytes to calculate delta (use later to update progress bar).

ekharkunov · 2024-03-18T15:12:33Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

                        return;
                    }
-                    currentAttempt = currentAttempt + 1;
+                    if (onretry) onretry(xhr, event, xhr._loadedSize, currentAttempt);


Add onretry callback. It uses to decrease loaded amount of bytes and update progress bar with correct values.

ekharkunov · 2024-03-18T15:13:36Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

-            }
+        request.onprogress = function(xhr, e, ls) {
+            var delta = e.loaded - ls;
+            onprogress(delta);


Simplify onprogress because we don't need to know total size from response headers (see #7039)

ekharkunov · 2024-03-18T15:13:57Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

-    wasm_size: 2000000,
-    wasmjs_size: 250000,
-    asmjs_size: 4000000,
+    wasm_size: {{ DEFOLD_WASM_SIZE }},


Bake final size of files inside dmloader.

ekharkunov · 2024-03-18T15:14:51Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

-    // load engine (asm.js or wasm.js + wasm)
+    // left as entrypoint for backward capability
+    // start loading archive_files.json
+    // after receiving it - start loading engine and data concurrently
    load: function(appCanvasId, exeName) {


I left EngineLoader.load function as entrypoint because it may produce breaking change for developer.

ekharkunov · 2024-03-18T15:15:19Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

@@ -241,9 +302,6 @@ var GameArchiveLoader = {
        this._onGameArchiveLoaderCompletedListeners = [];
        this._onAllTargetsBuiltListeners = [];
        this._onFileDownloadErrorListeners = [];
-
-        this._currentDownloadBytes = 0;


No need to calculate it separately.

ekharkunov · 2024-03-18T15:16:33Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

-        // calculate total download size of all files
-        for(var i=0; i<this._files.length; ++i) {
-            this._totalDownloadBytes += this._files[i].size;
+        var isWASMSupported = Module['isWASMSupported'];


Main change:
Now it starts loading game data and game engine concurrently after receiving archive_files.json. Here it calculates total amount of bytes that need to be downloaded.

ekharkunov · 2024-03-18T15:17:04Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

@@ -457,19 +520,6 @@ var Progress = {
    progress_id: "defold-progress",
    bar_id: "defold-progress-bar",

-    listeners: [],
-
-    addListener: function(callback) {


Left Progress object only for visualization.

It's not that critical if we break something in templates, because we always can notify users and ask them to update their template (btw could you please a it into your PR message e.g. like here) but it would be nice to keep existing extensions compatible if it's possible. For example:

https://github.com/defold/extension-poki-sdk/blob/main/poki-sdk/manifests/web/engine_template.html

If it's impossible to avoid - sure, we have to notify users and probably introduce it as a breaking change, but here it feels like something only about naming. I suggest to keep Progress for logic and create ProgressView just for you, to keep it back compatible

I left Progress object as wrapper around new ProgressView and ProgressUpdater. Let me know if such approach is ok. @AGulev

ekharkunov · 2024-03-18T15:18:08Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

@@ -499,6 +544,50 @@ var Progress = {
    }
 };

+var ProgressUpdater = {


Introduce new object ProgressUpdater. It should control and recalculate downloading progress and notify all listeners.

ekharkunov · 2024-03-18T15:18:39Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

@@ -856,6 +890,119 @@ var Module = {
    },
 };

+// common engine setup


Part of code that was moved from engine_template.html.

engine/engine/content/builtins/manifests/web/dark_theme.css

ekharkunov · 2024-03-18T15:19:19Z

engine/engine/content/builtins/manifests/web/light_theme.css

@@ -13,6 +13,7 @@
 		background-color: #1a72eb;
 		text-align: center;
 		line-height: 20px;
+		transition: width 1s ease;


For smoother progress bar width update.

ekharkunov · 2024-03-18T15:24:44Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js


-    stream_wasm: false,
+    updateWasmInstantiateProgress: function(totalDownloadedSize) {
+        EngineLoader.wasm_instantiate_progress = totalDownloadedSize * 0.1;


Calculate what amount of total downloadable size is 10% and use if as part of progress that indicates wasm instantiating.

… Add dmloader.js creation during bundle process. Cleanup some unused code from Bob. Rework Progress in dmloader. Add totlaSize field to archive_list.json. Add onretry calbback. Decrease downloaded amount of bytes if request failed and retried. Add easing for base progress bar width change.

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/HTML5Bundler.java

ekharkunov · 2024-03-18T22:06:47Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

@@ -510,6 +599,7 @@ var Module = {
    _archiveLoaded: false,
    _preLoadDone: false,
    _waitingForArchive: false,
+    _isEngineLoaded: false,


Introduce new flag to control if engine already downloaded or not. It's need to handle case when game data loaded faster than engine binaries.

ekharkunov · 2024-03-18T22:07:14Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

-            FileLoader.options.retryInterval = params["retry_time"] * 1000;
-            if (typeof params["can_not_download_file_callback"] === "function") {
-                GameArchiveLoader.addFileDownloadErrorListener(params["can_not_download_file_callback"]);
+            if (Module._archiveLoaded) {


If game data already loaded - call main function.

ekharkunov · 2024-03-18T22:08:22Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

-                Module.noInitialRun = false;
-            } else {
-                Module.callMain(Module.arguments);
+            if (Module._isEngineLoaded) {


Check if engine binary loaded and JS runtime is initialize (it means that 'onRuntimeInitialized' was called).

ekharkunov · 2024-03-18T22:09:18Z

Note to reviewers:
In the last commit implement case when game data loaded faster than engine binaries.

ekharkunov · 2024-03-18T22:41:40Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

        };
        request.onerror = function(xhr, e) {
            onerror("Error loading '" + url + "' (" + e + ")");
        };
        request.onload = function(xhr, e) {
-            if (xhr.readyState === 4) {
+            if (xhr.readyState === XMLHttpRequest.DONE) {


Use constant instead of magic number.

AGulev · 2024-03-19T07:47:29Z

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/resources/jsweb/dmloader.js

@@ -457,19 +520,6 @@ var Progress = {
    progress_id: "defold-progress",
    bar_id: "defold-progress-bar",

-    listeners: [],
-
-    addListener: function(callback) {


It's not that critical if we break something in templates, because we always can notify users and ask them to update their template (btw could you please a it into your PR message e.g. like here) but it would be nice to keep existing extensions compatible if it's possible. For example:

https://github.com/defold/extension-poki-sdk/blob/main/poki-sdk/manifests/web/engine_template.html

If it's impossible to avoid - sure, we have to notify users and probably introduce it as a breaking change, but here it feels like something only about naming. I suggest to keep Progress for logic and create ProgressView just for you, to keep it back compatible

engine/engine/content/builtins/manifests/web/dark_theme.css

engine/engine/content/builtins/manifests/web/engine_template.html

AGulev

LGTM!👍

britzl

Looks good! 👍

ekharkunov commented Mar 18, 2024

View reviewed changes

engine/engine/content/builtins/manifests/web/dark_theme.css Outdated Show resolved Hide resolved

ekharkunov commented Mar 18, 2024

View reviewed changes

ekharkunov force-pushed the issue-7456-web-loading-parallel branch from efb37f1 to 53beeb0 Compare March 18, 2024 15:26

ekharkunov marked this pull request as ready for review March 18, 2024 15:26

ekharkunov marked this pull request as draft March 18, 2024 17:12

britzl reviewed Mar 18, 2024

View reviewed changes

com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/bundle/HTML5Bundler.java Show resolved Hide resolved

Handle case when data loads faster than engine.

97ea20e

ekharkunov commented Mar 18, 2024

View reviewed changes

ekharkunov marked this pull request as ready for review March 18, 2024 22:08

ekharkunov commented Mar 18, 2024

View reviewed changes

Cleanup unused code from dmloader.js.

27dc415

AGulev reviewed Mar 19, 2024

View reviewed changes

ekharkunov added 2 commits March 25, 2024 15:04

Rename Progress variables for backward compatability.

914dca2

Rework progressbar animation with trasform property.

e1e83fb

ekharkunov mentioned this pull request Mar 25, 2024

Update html5 manual with new information about CUSTOM_PARAMETERS. defold/doc#413

Merged

ekharkunov requested review from britzl and AGulev March 26, 2024 09:54

AGulev approved these changes Mar 26, 2024

View reviewed changes

britzl approved these changes Mar 26, 2024

View reviewed changes

ekharkunov merged commit f41dffa into defold:dev Mar 26, 2024
1 check passed

ekharkunov mentioned this pull request Apr 9, 2024

Replace Emscripten API (Emscripten 3.1.55). defold/extension-kaios#10

Merged

britzl changed the title ~~[HTML5] Load engine and data concurrently~~ Load engine and data concurrently Apr 10, 2024

aglitchman mentioned this pull request Apr 25, 2024

HTML5: Engine starts twice and game freezes #8848

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load engine and data concurrently #8682

Load engine and data concurrently #8682

ekharkunov commented Mar 18, 2024 •

edited

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

AGulev Mar 19, 2024

ekharkunov Mar 25, 2024 •

edited

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov Mar 18, 2024

ekharkunov commented Mar 18, 2024

ekharkunov Mar 18, 2024

AGulev Mar 19, 2024

AGulev left a comment

britzl left a comment

		generator.writeEndArray();
		generator.writeNumberField("total_size", totalSize);

Load engine and data concurrently #8682

Load engine and data concurrently #8682

Conversation

ekharkunov commented Mar 18, 2024 • edited

Technical changes

PR checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekharkunov Mar 25, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekharkunov commented Mar 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AGulev left a comment

Choose a reason for hiding this comment

britzl left a comment

Choose a reason for hiding this comment

ekharkunov commented Mar 18, 2024 •

edited

ekharkunov Mar 25, 2024 •

edited