Restructuring emscripten_dlopen_js #21595

impact-maker · 2024-03-22T13:48:27Z

Solves #21440
emscripten_dlopen_js

Description:

Promise-Based Approach as the cannonical approach:
Anchoring asynchronous operations within a Promise object, giving a method to manage asynchronous tasks

Callback Support: Maintains backward compatibility with callback functions (onsuccess, onerror)

ASYNCIFY Integration: Bridges synchronous C/C++ code with asynchronous JavaScript operations by compiling the WebAssembly module with ASYNCIFY enabled and specifying asynchronous functions like _emscripten_dlopen_js in ASYNCIFY_IMPORTS

impact-maker · 2024-03-22T13:51:23Z

@sbc100 @tlively I have made an initial api restructuring draft for emscripten_dlopen_js async function. Kindly provide feedback on this implementation

sbc100 · 2024-03-22T15:25:22Z

src/library_dylink.js

    }

-    {{{ runtimeKeepalivePush() }}}
+    runtimeKeepalivePush();


I think you need to keep the {{{ and }}} around these calls.

Emscripten's JS is run though a preprocessing step that gathers all the libraries and resolves various macros before emitting the final JS. The triple curly braces {{{}}} tell the preprocessor to replace this code fragment with the result of running the code inside the braces at build time.

sbc100 · 2024-03-22T15:26:36Z

src/library_dylink.js

    var promise = dlopenInternal(handle, { loadAsync: true });
    if (promise) {
      promise.then(successCallback, errorCallback);
    } else {
-      errorCallback();
+      errorCallback(new Error('Promise was not created by dlopenInternal.'));


Instead of this if/elfse here I think we can just asset(promise) here, but that sounds like a separate PR that we can land first.

sbc100 · 2024-03-22T15:36:10Z

src/library_dylink.js

    }

-    {{{ runtimeKeepalivePush() }}}
+    runtimeKeepalivePush();
    var promise = dlopenInternal(handle, { loadAsync: true });


So this is an example of API that already has a promise being returned from the underlying API.

Do I think we have two reasonable ways forward to make this more generic:

Return a promise_t back to C/C++ here and let C/C++ handler wrapping this into a callback-based API.

Create a generic JS wrapper function called something like promiseToCallback(successFnPtr, errorFrPtr) that takes care of doing all the runtimeKeepalivePop stuff and the callUserCallback stuff.

I think (2) might be the simplest for now as it probably involves a little less complexity since I believe the C/C++ promise API would end up creating an extra promise in order to do the emscripten_promise_then operation.

@tlively would be OK with (1) here?

Did you mean (2)? Either sounds good to me 👍

@sbc100 @tlively Made the changes according to the feedback provided above

Created a generic JS wrapper function called something like promiseToCallback(successFnPtr, errorFrPtr) that takes care of doing all the runtimeKeepalivePop stuff and the callUserCallback

Added {{{ and }}} around calls.

Added asset(promise) instead of if/elfse

sbc100 · 2024-03-26T22:39:08Z

src/library_dylink.js

-      {{{ runtimeKeepalivePop() }}}
-      callUserCallback(() => {{{ makeDynCall('vpp', 'onerror') }}}(handle, user_data));
+  // Utility function to encapsulate promise handling with callbacks 
+function promiseToCallback(promise, successFnPtr, errorFnPtr, user_data) {


You need to use the $promiseToCallback: function(...) { syntax here since this is inside a JS object.

You then need to add $promiseToCallback to the __deps for _emscripten_dlopen_js

tlively · 2024-03-28T16:54:37Z

src/library_dylink.js

-      var filename = UTF8ToString(handle + {{{ C_STRUCTS.dso.name }}});
-      dlSetError(`'Could not load dynamic lib: ${filename}\n${e}`);
-      {{{ runtimeKeepalivePop() }}}


It looks like the {{{ }}} are still removed from this. Were you able to run the tests locally? They should be failing because of this problem.

@tlively necessary changes were missed in the previous commit. I have now pushed the latest changes. I have gone through the documentation but still doubtful on what commands and test suites needs to run to confirm for the api restructuring changes.

Looking at CI, it looks one the failing tests is test_pthread_dlopen_many in the core0 suite. To run just that test locally, you would run test/runner core0.test_pthread_dlopen_many. To run all the tests in core0 (which means the core tests with optimization level O0), you would run test/runner core0. You can also use the --verbose and --save-dir flags to get more information and to save the files so you can manually run and debug the test yourself.

impact-maker · 2024-03-29T19:37:35Z

@tlively @sbc100 I have done the whole setup for test suite and pushed the code.
But after running the test , it is getting stuck
test/runner core0.test_pthread_dlopen_many
Test suites:
['test_core']
Running test_core: (1 tests)
(checking sanity from test runner)
shared:INFO: (Emscripten: Running sanity checks)
test_pthread_dlopen_many (test_core.core0.test_pthread_dlopen_many) ...

tlively · 2024-03-29T19:52:11Z

If you add --verbose to the runner command, it should print out the commands it's using to build and run the test case. You can then run those commands yourself to debug the code.

impact-maker · 2024-04-15T17:30:47Z

@tlively @sbc100 @kripken I have made a revision in dlopen api which inverts the callback functionality to use promises as the main form and implementating callbacks and asyncify around it. Can you provide me feedback on this so that I can take it ahead from here.

sbc100 · 2024-04-15T22:11:53Z

src/library_dylink.js

-  _emscripten_dlopen_js: (handle, onsuccess, onerror, user_data) => {
-    /** @param {Object=} e */
+  // Utility function to encapsulate promise handling with callbacks
+  $promiseToCallback: function(promise, successFnPtr, errorFnPtr, user_data) {


I don't see this utility being called anywhere

sbc100 · 2024-04-15T22:12:13Z

src/library_dylink.js

-      errorCallback();
-    }
+    // Assert that the promise is provided
+    assert(promise, 'promiseToCallback was called without a valid promise.');


Put this line at the top of the function and wrap in #if ASSERTIONS check

sbc100 · 2024-04-15T22:14:05Z

system/lib/libc/dynlink.c

+
+  em_promise_t promise = emscripten_dlopen_promise(filename,flags);
+  emscripten_promise_then(promise, onsuccess, user_data);
+  emscripten_promise_catch(promise, onerror, user_data);


This nice, it seems like you have implemented something like promiseToCallback in native code?

I think it might be fast it if was implemented JS though since using promises from the native does have some amount of overhead I think (just calling back and forth into JS has a cost).

sbc100 · 2024-04-15T22:19:42Z

system/lib/libc/dynlink.c

-  }
-  do_write_lock();
-  char buf[2*NAME_MAX+2];
-  filename = resolve_path(buf, filename, sizeof buf);


What happened to this call to resolve_path?

sbc100 · 2024-04-15T22:20:40Z

src/library_dylink.js

+
+     {{{ runtimeKeepalivePush() }}}
+
+    var filename = UTF8ToString(handle);


If the first argument of this function now a filename and not a handle then perhaps it should be renamed.

sbc100 · 2024-04-15T22:21:45Z

src/library_dylink.js

  },

+  _emscripten_dlopen_js__deps: ['$dlopenInternal', '$runtimeKeepalivePush', '$runtimeKeepalivePop', '$dlSetError', '$dynCall'],
+  _emscripten_dlopen_js: function(handle, promise, onsuccess, onerror,) {


It seems a little odd to mix passing a promise here with passing callback. I would assume that we would want one or the other and not both.

In fact it looks like this code doesn't actually use the promise argument right now?

sbc100 reviewed Mar 22, 2024

View reviewed changes

sbc100 reviewed Mar 26, 2024

View reviewed changes

tlively reviewed Mar 28, 2024

View reviewed changes

impact-maker force-pushed the emscripten_dlopen_js branch from ff05732 to b7dbaa9 Compare March 28, 2024 17:50

impact-maker force-pushed the emscripten_dlopen_js branch from 435e835 to 09dcf70 Compare March 29, 2024 22:47

impact-maker added 3 commits March 30, 2024 21:41

restructuring emscripten_dlopen_js

4aa494d

restructed the code

5877e57

changed promise to callback format

2d2f41f

impact-maker force-pushed the emscripten_dlopen_js branch from 5a25b4b to 231135e Compare March 30, 2024 16:13

impact-maker marked this pull request as draft April 5, 2024 10:22

dlopen revision

de9e66b

impact-maker force-pushed the emscripten_dlopen_js branch from ca3932f to de9e66b Compare April 15, 2024 17:25

impact-maker marked this pull request as ready for review April 15, 2024 17:26

sbc100 reviewed Apr 15, 2024

View reviewed changes

added promisetoCallbacks and removed error

6d3a264

impact-maker requested review from tlively and sbc100 April 18, 2024 21:39

internal function restored

0508b17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restructuring emscripten_dlopen_js #21595

Restructuring emscripten_dlopen_js #21595

impact-maker commented Mar 22, 2024 •

edited

impact-maker commented Mar 22, 2024

sbc100 Mar 22, 2024

tlively Mar 22, 2024

sbc100 Mar 22, 2024

sbc100 Mar 22, 2024

tlively Mar 22, 2024

impact-maker Mar 23, 2024

sbc100 Mar 26, 2024

tlively Mar 28, 2024

impact-maker Mar 28, 2024

tlively Mar 28, 2024

impact-maker commented Mar 29, 2024

tlively commented Mar 29, 2024

impact-maker commented Apr 15, 2024 •

edited

sbc100 Apr 15, 2024

sbc100 Apr 15, 2024

sbc100 Apr 15, 2024

sbc100 Apr 15, 2024

sbc100 Apr 15, 2024

sbc100 Apr 15, 2024


		{{{ runtimeKeepalivePush() }}}

		var filename = UTF8ToString(handle);

Restructuring emscripten_dlopen_js #21595

Are you sure you want to change the base?

Restructuring emscripten_dlopen_js #21595

Conversation

impact-maker commented Mar 22, 2024 • edited

impact-maker commented Mar 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

impact-maker commented Mar 29, 2024

tlively commented Mar 29, 2024

impact-maker commented Apr 15, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

impact-maker commented Mar 22, 2024 •

edited

impact-maker commented Apr 15, 2024 •

edited