Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Latest Emscripten 2.0.33 works well, except file loading. [solved] #6781

Closed
Jonathhhan opened this issue Nov 12, 2021 · 91 comments
Closed

Latest Emscripten 2.0.33 works well, except file loading. [solved] #6781

Jonathhhan opened this issue Nov 12, 2021 · 91 comments

Comments

@Jonathhhan
Copy link
Contributor

Jonathhhan commented Nov 12, 2021

It was possible to compile with the latest Emscripten, for that I had to recompile the OF libs (FreeImage or freetype for example) with the current Emscripten and made some other small changes like using c++17 and std filesystem instead of boost (boost had an undefined symbol).
I guess the issue now has to do with the glm qualifier glm::qualifier)0>&, this is the Java Script error message for the imageSubsectionExample (everything compiles fine):
imageSubsectionExample.html:1 emscripten_set_main_loop_timing: Cannot set timing mode for main loop since a main loop does not exist! Call emscripten_set_main_loop first to set one up. printErr @ imageSubsectionExample.html:1 imageSubsectionExample.html:1 Aborted(native code called abort()) printErr @ imageSubsectionExample.html:1 imageSubsectionExample.js:1 Uncaught RuntimeError: Aborted(native code called abort()) at abort (imageSubsectionExample.js:1:34496) at _abort (imageSubsectionExample.js:1:119651) at std::__2::__throw_failure(char const*) (imageSubsectionExample.wasm:0x362df2) at std::__2::__fs::filesystem::__canonical(std::__2::__fs::filesystem::path const&, std::__2::error_code*) (imageSubsectionExample.wasm:0x380f6f) at ofToDataPath(std::__2::__fs::filesystem::path const&, bool) (imageSubsectionExample.wasm:0x55a2d) at ofApp::setup() (imageSubsectionExample.wasm:0x22653) at ofNode::onParentOrientationChanged(glm::qua<float, (glm::qualifier)0>&) (imageSubsectionExample.wasm:0x4638a) at std::__2::__function::__func<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&), std::__2::allocator<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&)>, bool (void const*, ofKeyEventArgs&)>::operator()(void const*&&, ofKeyEventArgs&) (imageSubsectionExample.wasm:0x4c0d6) at ofEvent<ofHttpResponse, std::__2::recursive_mutex>::notify(ofHttpResponse&) (imageSubsectionExample.wasm:0x2a046) at ofCoreEvents::notifySetup() (imageSubsectionExample.wasm:0x4cb73)
Is there a way to solve whether the boost, the glm or the std filesystem issue, so that is possible to load data with OF and the latest Emscripten (everything else seems to work fine)?

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Nov 12, 2021

This is the Java Script error message if I compile with boost (even if there is no file to load):

advanced3dExample.html:1 missing function: 
_ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC1ERKS5_
printErr @ advanced3dExample.html:1
advanced3dExample.html:1 Aborted(-1)
printErr @ advanced3dExample.html:1
advanced3dExample.js:1 Uncaught RuntimeError: Aborted(-1)
    at abort (advanced3dExample.js:1:34320)
    at __ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC1ERKS5_ (advanced3dExample.js:1:47694)
    at boost::filesystem::absolute(boost::filesystem::path const&, boost::filesystem::path const&) (advanced3dExample.wasm:0x9716b)
    at main (advanced3dExample.wasm:0x1c5b5)
    at advanced3dExample.js:1:35039
    at callMain (advanced3dExample.js:1:277462)
    at doRun (advanced3dExample.js:1:278046)
    at advanced3dExample.js:1:278201

And interestingly, if I compile with std filesystem and -O1 instead of -O3 the glm error is gone (which enables for example the polygonExample to work):

imageSubsectionExample.html:1 emscripten_set_main_loop_timing: Cannot set timing mode for main loop since a main loop does not exist! Call emscripten_set_main_loop first to set one up. printErr @ imageSubsectionExample.html:1 imageSubsectionExample.html:1 Aborted(native code called abort()) printErr @ imageSubsectionExample.html:1 imageSubsectionExample.js:201 Uncaught RuntimeError: Aborted(native code called abort()) at abort (imageSubsectionExample.js:1596:10) at _abort (imageSubsectionExample.js:5377:2) at void std::__2::__fs::filesystem::__throw_filesystem_error<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >&, std::__2::__fs::filesystem::path const&, std::__2::__fs::filesystem::path const&, std::__2::error_code const&>(std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >&, std::__2::__fs::filesystem::path const&, std::__2::__fs::filesystem::path const&, std::__2::error_code const&) (imageSubsectionExample.wasm:0x3d5671) at std::__2::__fs::filesystem::detail::(anonymous namespace)::ErrorHandler<std::__2::__fs::filesystem::path>::report(std::__2::error_code const&) const (imageSubsectionExample.wasm:0x3d561a) at std::__2::__fs::filesystem::__canonical(std::__2::__fs::filesystem::path const&, std::__2::error_code*) (imageSubsectionExample.wasm:0x3d54df) at ofToDataPath(std::__2::__fs::filesystem::path const&, bool) (imageSubsectionExample.wasm:0x4d9d7) at ofLoadImage(ofPixels_<unsigned char>&, std::__2::__fs::filesystem::path const&, ofImageLoadSettings const&) (imageSubsectionExample.wasm:0x10179) at ofImage_<unsigned char>::load(std::__2::__fs::filesystem::path const&, ofImageLoadSettings const&) (imageSubsectionExample.wasm:0x10a8c) at ofApp::setup() (imageSubsectionExample.wasm:0x9248) at ofBaseApp::setup(ofEventArgs&) (imageSubsectionExample.wasm:0x330e2)
So my question is: Is there is a way to make any of the filesystems work with this Emscripten version?

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Nov 12, 2021

I comiled boost again, and now OF works with boost and Emscripten 2.0.33...
Can add what I changed exactly for that, but basically I recompiled the necessary OF libs with Apothecary Emscripten 2.0.33.
Edit: Also had to change some other stuff, but now it seems to work like the last versions...

@Jonathhhan Jonathhhan changed the title Latest Emscripten 2.0.32 works well, except file loading. Latest Emscripten 2.0.32 works well, except file loading. [solved] Nov 12, 2021
@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Nov 13, 2021

Those are the additional changes (these changes need to be added, too: #6758):

library_html5video.js (remove type)

html5video_player_pixel_format: function(id){
    return allocate(intArrayFromString(VIDEO.players[id].pixelFormat), ALLOC_STACK);
},

html5video_grabber_pixel_format: function(id){
    return allocate(intArrayFromString(VIDEO.grabbers[id].pixelFormat), ALLOC_STACK);
},

index.html (keyevents)

<canvas class=emscripten id=canvas oncontextmenu=event.preventDefault() tabindex=-1></canvas>

library_html5.js(emsdk remove __ from getBoundingClientRect)

    #if !DISABLE_DEPRECATED_FIND_EVENT_TARGET_BEHAVIOR
    if (Module['canvas']) {
      var rect = getBoundingClientRect(Module['canvas']);
      HEAP32[idx + {{{ C_STRUCTS.EmscriptenMouseEvent.canvasX / 4 }}}] = (e.clientX - rect.left) * (Module['canvas'].width / rect.width);
      HEAP32[idx + {{{ C_STRUCTS.EmscriptenMouseEvent.canvasY / 4 }}}] = (e.clientY - rect.top) * (Module['canvas'].height / rect.height);
    } else { // Canvas is not initialized, return 0.
      HEAP32[idx + {{{ C_STRUCTS.EmscriptenMouseEvent.canvasX / 4 }}}] = 0;
      HEAP32[idx + {{{ C_STRUCTS.EmscriptenMouseEvent.canvasY / 4 }}}] = 0;
    }
#endif
    var rect = getBoundingClientRect(target);
    HEAP32[idx + {{{ C_STRUCTS.EmscriptenMouseEvent.targetX / 4 }}}] = (e.clientX - rect.left) * (target.width / rect.width);
    HEAP32[idx + {{{ C_STRUCTS.EmscriptenMouseEvent.targetY / 4 }}}] = (e.clientY - rect.top) * (target.height / rect.height);

config.emscripten.default.make (add DYNCALLS=1 and BINARYEN_EXTRA_PASSES=--one-caller-inline-max-function-size=1)

PLATFORM_LDFLAGS = -Wl,--gc-sections --preload-file bin/data@data --emrun -s DYNCALLS=1 -s MAX_WEBGL_VERSION=2 -s WEBGL2_BACKWARDS_COMPATIBILITY_EMULATION=1 -s BINARYEN_EXTRA_PASSES=--one-caller-inline-max-function-size=1

@Jonathhhan Jonathhhan changed the title Latest Emscripten 2.0.32 works well, except file loading. [solved] Latest Emscripten 2.0.33 works well, except file loading. [solved] Nov 13, 2021
@roymacdonald
Copy link
Member

hey @Jonathhhan Thanks for all this.
Can you push to your OF fork all this changes so it is easier to check it and not having to go through all this issue and manually changing stuff.?

@Jonathhhan
Copy link
Contributor Author

@roymacdonald yes, of course. I will do that the next days.

@roymacdonald
Copy link
Member

after installing emscripten 2.0.33 I cant compile as it complains the following:

error: undefined symbol: _ZN24ofxEmscriptenSoundStreamC1Ev (referenced by top-level compiled C/C++ code)
error: undefined symbol: tessDeleteTess (referenced by top-level compiled C/C++ code)
error: undefined symbol: tessNewTess (referenced by top-level compiled C/C++ code)

@Jonathhhan
it is not seeing ofxEmscriptenSoundStream. did you add it somewhere? the other errors are for tess2.
thanks

@roymacdonald
Copy link
Member

I was able to get it to compile. I mostly needed to recompile the core lib dependencies.

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Nov 21, 2021

@roymacdonald thats nice. It would be great if you can help me to get the ofxPd examples to run with Emscripten (for using it with AudioWorklets). ofxPd works for me as part of ofxOfelia with Emscripten and standalone, but somehow not with Emscripten without ofxOfelia...

@roymacdonald
Copy link
Member

Sure, no problem.
Let me get it to run the window properly. Did you manage to get GLES 2 shaders to work? I have not. I am modifying ofxAppEmscriptenWindow to make it work properly based on some examples that emscripten has.
once I get that to work it my guess is that it shouldnt be much trouble to get audio to work as I know very well how audio works within OF.

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Nov 21, 2021

Yes, GLES2 is working now (my example https://gameoflife3d.handmadeproductions.de/ actually uses shaders for the grid). I think its part of my changes.
#6758

changing this:

EGLint contextAttribs[] = { EGL_CONTEXT_MAJOR_VERSION, 3, EGL_NONE, EGL_NONE };
std::vector attribList =
{
EGL_RED_SIZE, EGL_DONT_CARE,
EGL_GREEN_SIZE, EGL_DONT_CARE,
EGL_BLUE_SIZE, EGL_DONT_CARE,
EGL_ALPHA_SIZE, EGL_DONT_CARE,
EGL_DEPTH_SIZE, EGL_DONT_CARE,
EGL_STENCIL_SIZE, EGL_DONT_CARE,
EGL_SAMPLE_BUFFERS, EGL_DONT_CARE,
EGL_RENDERABLE_TYPE, EGL_OPENGL_ES3_BIT_KHR,
EGL_NONE
};

in https://github.com/Jonathhhan/openFrameworks/blob/master/addons/ofxEmscripten/src/ofxAppEmscriptenWindow.cpp and adding the flag -s USE_WEBGL2=1 to config.emscripten.default.mk enables webgl2 without editing any Emscripten file (in addition I use -s WEBGL2_BACKWARDS_COMPATIBILITY_EMULATION=1 for backwards compatibility with webgl1 https://emscripten.org/docs/optimizing/Optimizing-WebGL.html).

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Nov 21, 2021

For AudioWorklets and SharedArraybuffer all used libs (for most things boost is enough) and also OF need to be compiled with -s USE_PTHREAD=1 or multi thread support. The pthread compiled libs work also if pthread isnt used.

@roymacdonald
Copy link
Member

@Jonathhhan I tried out your changes and cloned your OF repo, but it was not working. Maybe I missed something.
Although I got it to work by changing a lot of other things and making it rely more on the emscripten functions. Maybe I messed up something when trying to get it to compile. I will go back and start over now that I have the necesary libs compiled and working. I will let you know how it goes. c
cheers

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Nov 21, 2021

@roymacdonald sorry, I dont have all the changes in my repo yet. I will tell you, if so. This is the mentioned and used Emscripten AudioWorklet branch: https://github.com/tklajnscek/emscripten/tree/worklet_support.

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Nov 21, 2021

@roymacdonald I made an OF AudioWorklet branch (I hope, its fine like that) that includes all updates for Emscripten 2.0.34 (and some other fixes) and for the Emscripten AudioWorklet branch: https://github.com/Jonathhhan/openFrameworks/tree/AudioWorklet
Only change that needs to be done in Emscripten is replacing file-packager.py with this file: https://github.com/Jonathhhan/emscripten/blob/master/tools/file_packager.py (basically I just added

if (typeof window === "object")

at line 291)

@roymacdonald
Copy link
Member

@Jonathhhan awesome! I'll check it and get back with comments (and see how to make all the other audio stuff to work)

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Nov 21, 2021

This is the error message with the pdExample from ofxPd:

pdExample.html:1 emscripten_set_main_loop_timing: Cannot set timing mode for main loop since a main loop does not exist! Call emscripten_set_main_loop first to set one up.
printErr @ pdExample.html:1
err @ pdExample.js:1
_emscripten_set_main_loop_timing @ pdExample.js:1
_eglSwapInterval @ pdExample.js:1
$ofxAppEmscriptenWindow::setVerticalSync(bool) @ pdExample.wasm:0x1ca18
$ofSetVerticalSync(bool) @ pdExample.wasm:0xfd379
$ofApp::setup() @ pdExample.wasm:0x13f35
$ofNode::onParentOrientationChanged(glm::qua<float, (glm::qualifier)0>&) @ pdExample.wasm:0xfa4ea
$std::__2::__function::__func<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&), std::__2::allocator<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&)>, bool (void const*, ofKeyEventArgs&)>::operator()(void const*&&, ofKeyEventArgs&) @ pdExample.wasm:0x101ac5
$ofEvent<ofHttpResponse, std::__2::recursive_mutex>::notify(ofHttpResponse&) @ pdExample.wasm:0x1d2c6
$ofCoreEvents::notifySetup() @ pdExample.wasm:0x102dbd
$ofxAppEmscriptenWindow::loop() @ pdExample.wasm:0x1c299
$std::__2::__function::__func<void (*)(), std::__2::allocator<void (*)()>, void ()>::operator()() @ pdExample.wasm:0x101255
$ofMainLoop::loop() @ pdExample.wasm:0x100d22
$ofRunApp(ofBaseApp*) @ pdExample.wasm:0xfd010
$__original_main @ pdExample.wasm:0x13e0e
$main @ pdExample.wasm:0x13eba
(anonymous) @ pdExample.js:1
callMain @ pdExample.js:1
doRun @ pdExample.js:1
(anonymous) @ pdExample.js:1
setTimeout (async)
run @ pdExample.js:1
runCaller @ pdExample.js:1
removeRunDependency @ pdExample.js:1
receiveInstance @ pdExample.js:1
receiveInstantiationResult @ pdExample.js:1
Promise.then (async)
(anonymous) @ pdExample.js:1
Promise.then (async)
instantiateAsync @ pdExample.js:1
createWasm @ pdExample.js:1
(anonymous) @ pdExample.js:1
pdExample.html:1 /
pdExample.html:1 Buffer size: 512
pdExample.html:1 
pdExample.html:1 BEGIN Patch Test
pdExample.js:1 Uncaught RuntimeError: null function or function signature mismatch
    at pd_typedmess (pdExample.wasm:0x58c64)
    at binbuf_eval (pdExample.wasm:0x30f3b)
    at canvas_objtext (pdExample.wasm:0xaea1c)
    at canvas_obj (pdExample.wasm:0xae8d1)
    at pd_typedmess (pdExample.wasm:0x58720)
    at binbuf_eval (pdExample.wasm:0x30f3b)
    at binbuf_evalfile (pdExample.wasm:0x341a3)
    at glob_evalfile (pdExample.wasm:0x2e003)
    at libpd_openfile (pdExample.wasm:0x280eb)
    at ofxPd::openPatch(std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&) (pdExample.wasm:0x206b3)
$pd_typedmess @ pdExample.wasm:0x58c64
$binbuf_eval @ pdExample.wasm:0x30f3b
$canvas_objtext @ pdExample.wasm:0xaea1c
$canvas_obj @ pdExample.wasm:0xae8d1
$pd_typedmess @ pdExample.wasm:0x58720
$binbuf_eval @ pdExample.wasm:0x30f3b
$binbuf_evalfile @ pdExample.wasm:0x341a3
$glob_evalfile @ pdExample.wasm:0x2e003
$libpd_openfile @ pdExample.wasm:0x280eb
$ofxPd::openPatch(std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&) @ pdExample.wasm:0x206b3
$ofApp::setup() @ pdExample.wasm:0x14430
$ofNode::onParentOrientationChanged(glm::qua<float, (glm::qualifier)0>&) @ pdExample.wasm:0xfa4ea
$std::__2::__function::__func<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&), std::__2::allocator<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&)>, bool (void const*, ofKeyEventArgs&)>::operator()(void const*&&, ofKeyEventArgs&) @ pdExample.wasm:0x101ac5
$ofEvent<ofHttpResponse, std::__2::recursive_mutex>::notify(ofHttpResponse&) @ pdExample.wasm:0x1d2c6
$ofCoreEvents::notifySetup() @ pdExample.wasm:0x102dbd
$ofxAppEmscriptenWindow::loop() @ pdExample.wasm:0x1c299
$std::__2::__function::__func<void (*)(), std::__2::allocator<void (*)()>, void ()>::operator()() @ pdExample.wasm:0x101255
$ofMainLoop::loop() @ pdExample.wasm:0x100d22
$ofRunApp(ofBaseApp*) @ pdExample.wasm:0xfd010
$__original_main @ pdExample.wasm:0x13e0e
$main @ pdExample.wasm:0x13eba
(anonymous) @ pdExample.js:1
callMain @ pdExample.js:1
doRun @ pdExample.js:1
(anonymous) @ pdExample.js:1
setTimeout (async)
run @ pdExample.js:1
runCaller @ pdExample.js:1
removeRunDependency @ pdExample.js:1
receiveInstance @ pdExample.js:1
receiveInstantiationResult @ pdExample.js:1
Promise.then (async)
(anonymous) @ pdExample.js:1
Promise.then (async)
instantiateAsync @ pdExample.js:1
createWasm @ pdExample.js:1
(anonymous) @ pdExample.js:1
pdExample.js:1 Uncaught TypeError: Cannot read properties of undefined (reading 'slice')
    at pdExample.js:1:5077
(anonymous) @ pdExample.js:1

@roymacdonald
Copy link
Member

I got that error before and fixed it.
And I ended up doing some massive changes to ofxAppEmscriptenWindow. I think that now it looks better. :)
https://github.com/roymacdonald/openFrameworks/blob/emscriptenUpdate/addons/ofxEmscripten/src/ofxAppEmscriptenWindow.cpp
https://github.com/roymacdonald/openFrameworks/blob/emscriptenUpdate/addons/ofxEmscripten/src/ofxAppEmscriptenWindow.h

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Nov 21, 2021

@roymacdonald with your changes I get kind of a similar error as in the pdExample (but only one of them):

EmscriptenExample.js:1 Uncaught TypeError: Cannot read properties of undefined (reading 'slice')
    at EmscriptenExample.js:1:5966
(anonymous) @ EmscriptenExample.js:1

while undefined is Module["preRun"]:

var necessaryPreJSTasks = Module["preRun"].slice();
if (typeof window === "object") {
    Module["arguments"] = window.location.search.substr(1).trim().split("&");
    if (!Module["arguments"][0]) {
        Module["arguments"] = []
    }
}
if (!Module["preRun"])
    throw "Module.preRun should exist because file support used it; did a pre-js delete it?";
necessaryPreJSTasks.forEach(function(task) {
    if (Module["preRun"].indexOf(task) < 0)
        throw "All preRun tasks that exist before user pre-js code should remain after; did you replace Module or modify Module.preRun?"
});
var moduleOverrides = {};
var key;
for (key in Module) {
    if (Module.hasOwnProperty(key)) {
        moduleOverrides[key] = Module[key]
    }
}
var arguments_ = [];
var thisProgram = "./this.program";
var quit_ = function(status, toThrow) {
    throw toThrow
};

The patch loads fine, but without the audioWorklet...
But maybe it is possible to use -s PROXY_TO_PTHREAD=1 -s OFFSCREENCANVAS_SUPPORT=1 -s OFFSCREEN_FRAMEBUFFER=1 with your changes... :)

@roymacdonald
Copy link
Member

I am not even able to run the pd example as it fails compiling it, although the OF audio examples work fine. There are some issues with the ofxPd addon. Are you using a branch other than ofxPd's master?

@roymacdonald
Copy link
Member

btw, I am using macos. i understand you use linux, right?

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Nov 21, 2021

yes, ubuntu 20.04. I forgot to mention, I think I use the ofxPd lib that is included in ofxOfelia and I set the inputs to 0...
with a little bit of work it could be possible to use offscreen rendering with your changes (already have an image, but I cant resize it - and it conflicts with the audioworklet ;) )...

@Jonathhhan
Copy link
Contributor Author

It works with -s PROXY_TO_PTHREAD=1 -s OFFSCREENCANVAS_SUPPORT=1 -s OFFSCREEN_FRAMEBUFFER=1, sadly not yet together with audioworklet yet, but it seems to be very efficient.
But I have to set the canvas size in the html file, if I try to set it in java script I get this message:

Uncaught DOMException: Failed to set the 'width' property on 'HTMLCanvasElement': Cannot resize canvas after call to transferControlToOffscreen().
    at Object.updateCanvasDimensions (http://localhost:6931/EmscriptenExample.js:1:218824)
    at Object.setCanvasSize (http://localhost:6931/EmscriptenExample.js:1:217256)
    at _emscripten_set_canvas_size (http://localhost:6931/EmscriptenExample.js:1:223031)
    at _emscripten_receive_on_main_thread_js (http://localhost:6931/EmscriptenExample.js:1:220745)
    at _do_call (http://localhost:6931/EmscriptenExample.wasm:wasm-function[15107]:0x8a3324)
    at emscripten_current_thread_process_queued_calls (http://localhost:6931/EmscriptenExample.wasm:wasm-function[15105]:0x8a2b6b)
    at emscripten_main_thread_process_queued_calls (http://localhost:6931/EmscriptenExample.wasm:wasm-function[15117]:0x8a36c6)
    at http://localhost:6931/EmscriptenExample.js:1:29890
    at Worker.worker.onmessage (http://localhost:6931/EmscriptenExample.js:1:39637)

If I dont remove the audioworklet part with offscreen canvas I get this message (maybe because it is not possible to create a worker from a worker...):

EmscriptenExample.worker.js:1 worker.js onmessage() captured an uncaught exception: TypeError: Cannot read properties of undefined (reading 'audioWorklet')
self.onmessage @ EmscriptenExample.worker.js:1
EmscriptenExample.worker.js:1 TypeError: Cannot read properties of undefined (reading 'audioWorklet')
    at Object.initAudioWorkletPThread (EmscriptenExample.js:1484:27)
    at _html5audio_stream_create (EmscriptenExample.js:9687:13)
    at ofxEmscriptenSoundStream::setup(ofSoundStreamSettings const&) (EmscriptenExample.wasm:0x21b75)
    at ofxOfelia::init(int, int, int, int, bool, bool, int, int, bool, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&) (EmscriptenExample.wasm:0x28ae6)
    at ofApp::setup() (EmscriptenExample.wasm:0x1d57a)
    at ofNode::onParentOrientationChanged(glm::qua<float, (glm::qualifier)0>&) (EmscriptenExample.wasm:0x523af3)
    at std::__2::__function::__func<std::__2::shared_ptr<of::priv::Function<ofEventArgs, std::__2::recursive_mutex> > ofEvent<ofEventArgs, std::__2::recursive_mutex>::make_function<ofEasyCam>(ofEasyCam*, void (ofEasyCam::*)(ofEventArgs&), int)::'lambda'(void const*, ofEventArgs&), std::__2::allocator<std::__2::shared_ptr<of::priv::Function<ofEventArgs, std::__2::recursive_mutex> > ofEvent<ofEventArgs, std::__2::recursive_mutex>::make_function<ofEasyCam>(ofEasyCam*, void (ofEasyCam::*)(ofEventArgs&), int)::'lambda'(void const*, ofEventArgs&)>, bool (void const*, ofEventArgs&)>::operator()(void const*&&, ofEventArgs&) (EmscriptenExample.wasm:0x51c5d6)
    at ofEvent<ofHttpResponse, std::__2::recursive_mutex>::notify(ofHttpResponse&) (EmscriptenExample.wasm:0x2076c)
    at ofCoreEvents::notifySetup() (EmscriptenExample.wasm:0x533088)
    at ofxAppEmscriptenWindow::loop() (EmscriptenExample.wasm:0x1f6ab)

@Jonathhhan
Copy link
Contributor Author

And without -s PROXY_TO_PTHREAD=1 -s OFFSCREENCANVAS_SUPPORT=1 -s OFFSCREEN_FRAMEBUFFER=1 your updated files work with audioworklet if I remove:

var necessaryPreJSTasks = Module["preRun"].slice();
if (typeof window === "object") {
    Module["arguments"] = window.location.search.substr(1).trim().split("&");
    if (!Module["arguments"][0]) {
        Module["arguments"] = []
    }
}
if (!Module["preRun"])
    throw "Module.preRun should exist because file support used it; did a pre-js delete it?";
necessaryPreJSTasks.forEach(function(task) {
    if (Module["preRun"].indexOf(task) < 0)
        throw "All preRun tasks that exist before user pre-js code should remain after; did you replace Module or modify Module.preRun?"
});

from the generated java script file (at line 239).

@roymacdonald
Copy link
Member

Great!
it sounds that Module["preRun"] got deleted and thats is why it complains. I have not gotten ofxPd to run yet (haven't done much though).

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Nov 22, 2021

Here is an example with -s PROXY_TO_PTHREAD=1 -s OFFSCREENCANVAS_SUPPORT=1 -s OFFSCREEN_FRAMEBUFFER=1:
https://test2.handmadeproductions.de/
I would say it runs better than without (only the sound and resizing issues then).
Here for comparison without offscreen rendering: https://arturocastro.net/files/of-emscripten/assimpExample/
I also put alpha off, because it results in a white background with offscreen canvas rendering (or whatever color the background has). Offscreen canvas does not seem to work with firefox, but chrome and edge (thats what I tested):
pthread_create: failed to transfer control of canvas "canvas" to OffscreenCanvas! Error: [Exception... "Method not implemented" nsresult: "0x80004001 (NS_ERROR_NOT_IMPLEMENTED)" location: "JS frame :: https://test2.handmadeproductions.de/assimpExample.js :: ___pthread_create_js :: line 2209" data: no]

@roymacdonald
Copy link
Member

So I am able to run the sound examples for input and output audio with graphics and all. One thing I noticed was that the files were not being able to load because of ofToDataPath was not giving the correct path.
in ofUtils.cpp i modified the defaultDataPath function to look like this


    string defaultDataPath(){
    #if defined TARGET_OSX
        try{
            return std::filesystem::canonical(ofFilePath::join(ofFilePath::getCurrentExeDir(),  "../../../data/")).string();
        }catch(...){
            return ofFilePath::join(ofFilePath::getCurrentExeDir(),  "../../../data/");
        }
    #elif defined TARGET_ANDROID
        return string("sdcard/");
    #elif defined(TARGET_EMSCRIPTEN) /// added this
        return string("data/"); // and this to make it work with emscripten
    #else
        try{
            return std::filesystem::canonical(ofFilePath::join(ofFilePath::getCurrentExeDir(),  "data/")).make_preferred().string();
        }catch(...){
            return ofFilePath::join(ofFilePath::getCurrentExeDir(),  "data/");
        }
    #endif
    }

Now I can load files yet still ofxPd gives me odd errors which dont have much info in order to debug. So, out of this I would be inclined to say that the problem lies in either ofxPd or libpd, rather than in OF as audio works well.

@ofTheo
Copy link
Member

ofTheo commented Feb 2, 2022

Hey @Jonathhhan

Trying to get C++17 in via this PR:
#6844

-std=c++17 is getting past through to the compiler

/emsdk/upstream/emscripten/em++ -g0 -DDEBUG -Wall -std=c++17 

but I am running into this error:

/src/libs/openFrameworks/utils/ofFileUtils.cpp:650:12: error: use of undeclared identifier 'getegid'
229
        }else if (getegid() == info.st_gid){

Did you ever get OF working with C++17 and not using boost for the filesystem stuff?

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Feb 2, 2022

Hey @ofTheo,
if I remember right, then it worked for the graphicsExample, because there is no data that needs to be loaded.
But since most of the patches need data I gave up with that.
I changed a lot in the java script file for the ofxEmscriptenAudioPlayer. It loads instantly now, the issues are gone, and it needs much less memory. Not sure if I destroyed some of the functionalty, but it should work if it is done right. Basically it replaces the audioBuffer with MediaElementSource. Here is a (working) draft: https://github.com/Jonathhhan/ofEmscriptenExamples/blob/main/emscriptenAudioInput/library_html5audioWorklet.js

(Its also working without audioWorklets, just need to change 1 or 2 variables...)
Only drawback so far, before loading a new sound the old one needs to be unloaded with soundPlayer.unload() for stopping the old sound and revoking the url (maybe its possible to include that in load(), but I dont know how to get the old sound id there...).

Also updated the branch my example to show the changes: https://audiovideopdprocessing.handmadeproductions.de/

I thought about the urls that I create from JS: They maybe only work because they are send back and read from JS. And that works only for ofxEmscriptenVideoPlayer and ofxEmscriptenSoundPlayer (which is very nice, by the way...). Something like ofImage would need a different pointer (I am only guessing). Or I have to check outhow to load local files into the Emscripten filesystem and read from there (some help with that is kindly appreciated)...

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Feb 2, 2022

Hey @ofTheo and @roymacdonald, what do you think about my changes of the ofxEmscriptenAudioPlayer? Does it make sense?
Pan should also be implementable easily: https://stackoverflow.com/questions/20287890/audiocontext-panning-audio-of-playing-media

var ambientAudio = new Audio('./ambientVideo.mp4');

document.addEventListener('keydown', ambientAudioControl)
function ambientAudioControl(e) {
  if (e.keyCode === 37) panToLeft()
  if (e.keyCode === 39) panToRight()
  if (e.keyCode === 40) panToStereo()
}

const ambientContext = new AudioContext();
const source = ambientContext.createMediaElementSource(ambientAudio);
const ambientPan = ambientContext.createStereoPanner()

function panToLeft(){ ambientPan.pan.value = -1 }
function panToRight(){ ambientPan.pan.value = 1 }
function panToStereo(){ ambientPan.pan.value = 0 }

source.connect(ambientPan)
ambientPan.connect(ambientContext.destination)
ambientAudio.play()

Regarding unvoking the urls: Maybe also create an unload() method for ofxEmscriptenVideoPlayer for that?
Or better (but no idea how to do that): Unvoke the old url automatically if a new file is loaded...

And one idea regarding drag and drop: If it would be possible to define a html drop area from OF, then drag and drop would make sense (right now it works, but only for the whole window as the drop area).

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Feb 3, 2022

So I added pan for audio and video...

And I ask myself: Is it really necessary to create several audioContexts or would one be enough (I ask because it seems not possible to connect one node of one context to a node of another context, and for what I did one audioContext with id 0 was enough)?
And it would be easier then to connect the audio node from the video player to the audioContext.
I understand why I need several sound and video id's (one for every player) but not sure about the audioContext.
https://stackoverflow.com/questions/60306160/multiple-audio-contexts-whats-the-use-case

And a stream is created as soon as I have an ofApp::audioReceived() and / or an ofApp::audioRequested() method?

And one idea regarding connecting the audio and video player to the audiostream: They could get a method like audioPlayer.connectToStream() or something like that.
And for switching between the old audio stream method (scriptProcessor: https://developer.mozilla.org/en-US/docs/Web/API/BaseAudioContext/createScriptProcessor) and audioWorklet there could be a method like ofxEmscriptenSoundStream.connectToAudioWorklet()... (with scriptProcessor as the default)?
And one issue with sharedArrayBuffer and soundPlayer.getSystemSpectrum() (which wasnt really needed yet from my side - and is also doable with ofxPd): Uncaught TypeError: Failed to execute 'getFloatFrequencyData' on 'AnalyserNode': The provided Float32Array value must not be shared. (can be avoided with -s PTHREADS=0, but then audioWorklet does not work).

@roymacdonald
Copy link
Member

Hey @Jonathhhan
Thanks for doing all this.
I think taht the players, both video and audio should behave in the same way as the others do, so switching from one to the other makes no difference.
So I think that the pan function should be as in ofBaseSoundPlayer where you pass float to set the pan, instead of just having left or right.

If you do really need to use a sound stream to make it work then ofxEmscriptenSoundPlayer and ofxEmscriptenVideoPlayer should both inherit from `ofBaseSoundOutput'

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Feb 4, 2022

@roymacdonald you already set a float for pan, but from -1 to 1. I think in OF its 0 - 1? No problem to change that (I hope I understood it right).
See it here: https://audiovideopdprocessing.handmadeproductions.de/
What is more difficult (for me) is to set the right initial volume (default is 1), but with pan it gets easily distorted then.
Right now I set the initial volume too low, but its not distorted. I have to find the right value, or lower the volume if it gets panned (I guess the volumes of both channels are added at the moment). But it is also great, if it behaves like ofBaseSoundPlayer too.

ofxEmscriptenSoundPlayer and ofxEmscriptenVideoPlayer alone dont need a sound stream. I guess the soundstream is needed for audio input and for connecting the nodes from audio and video player to the input (so that I can process it with ofxPd ;) ), but maybe only if audioworklet is not used. not sure about that.

@roymacdonald
Copy link
Member

tha works nicely. I couldnt notice any distortion while moving the pan but I just used my laptop's speakers. No headphones atm.
pan is -1 to 1
this is what the documentation says /// \param pan range is -1 to 1 (-1 is full left, 1 is full right).

ofxEmscriptenSoundPlayer and ofxEmscriptenVideoPlayer alone dont need a sound stream. I guess the soundstream is needed for audio input and for connecting the nodes from audio and video player to the input (so that I can process it with ofxPd ;) ), but maybe only if audioworklet is not used. not sure about that.

I see. It depends on what you want to achieve if you need to use a soundstream or not. I am not sure what would be the best option here. Maybe a hybrid, which uses the sound stream if it is connected to it, otherwise it uses what you already implemented through the audio worklet?

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Feb 4, 2022

@roymacdonald Yes, I already lowered the initial volume, maybe a bit too much.
Another solution would be, to leave the initial value to 1 and lower it with setVolume when needed (like for panning a loud audio file).

Maybe a hybrid, which uses the sound stream if it is connected to it, otherwise it uses what you already implemented through the audio worklet?

That sounds good. The main problem with the audioWorklet is (in my opinion) that it relies on a non official Emscripten branch from @tklajnscek, and I dont know if it will be up to date. But we can ask the author and maybe he will update it for himself. He updated it for 3.0.1, not sure about the current and future versions. With this version it works great.
But I also think, if Emscripten audioWorklet will be released officially at some point, the OF implementation does not need to change much.

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Feb 7, 2022

I have some questions about the filesystem stuff:
It works now to load images, sound, text, etc. into the virtual filesystem, which is quite nice.
But I have the feeling, that it still makes sense to have .loadUrl(url); for the audio and video player, because it stays on the java script side (does not make sense to load it into the filesystem for that?). What do think about that? I also had a closer look at ofxEmscriptenURLFileLoader but I dont really know how to use it for my purpose (loading something as url into ofxEmscriptenAudioPlayer and ofxEmscriptenVideoPlayer) or in general.

@roymacdonald
Copy link
Member

roymacdonald commented Feb 7, 2022

Hey @Jonathhhan
At least from what I can see here it already would work with an URL.

I think there should not be a loadUrlfunction in these clases to be consistent with the rest of OF classes. For instance, take a look here to see how ofImage handles loading from either an URL or from a local file. I think that these classes you are working on should implement a similar thing.
It would be also a good idea to allow these to load from an ofBuffer object as well so async loading can be done.
Again, look here for how ofImage implements such.

ofxEmscriptenURLFileLoader is used internally by ofURLFileLoader file loader so you shouldn't need to use it directly, rather instead simply using ofURLFileLoader

Check this example https://github.com/openframeworks/openFrameworks/tree/master/examples/input_output/imageLoaderWebExample
I think that these classes should behave in the same way as ofImage does in it.

Let me know how it goes.
best

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Feb 7, 2022

Hey @roymacdonald thanks for the examples. that makes totally sense and would be much better than my solution. I will give it a try, but not sure if I am able implement to implement that into ofxEmscriptenAudioPlayer and ofxEmscriptenVideoPlayer.
The only additional methods from ofxEmscriptenAudioPlayer are getDurationSecs() and getDurationMS() (they were already there), but that is quite handy. And .setUsePixels() from ofxEmscriptenVideoPlayer which is really useful. And I added pan to the sound from ofxEmscriptenVideoPlayer not sure, if that makes sense...
Regarding images I wonder, if it is better to load them as an url (created from a local file) or to load them into the virtual filesystem (and delete them emmediatly after they are loaded into OF).

@roymacdonald
Copy link
Member

The only additional methods from ofxEmscriptenAudioPlayer are getDurationSecs() and getDurationMS() (they were already there), but that is quite handy. And .setUsePixels() from ofxEmscriptenVideoPlayer which is really useful. And I added pan to the sound from ofxEmscriptenVideoPlayer not sure, if that makes sense...

I just realized that ofSoundPlayer does not have a get duration method. which is quite strange that has not been added so far. I think that these should be getDuration() and getDurationMS(), where the former returns to seconds, this way it is consitent with ofVideoPLayer.

Let me know if you need help implementing ofxEmscriptenAudioPlayer or ofxEmscriptenVideoPlayer.
Keep on with this great contribution you are doing! Many thanks for it.
cheers

@ofTheo
Copy link
Member

ofTheo commented Feb 7, 2022

@roymacdonald would you have some time to help @Jonathhhan wrap up all the fairly clean emscripten fixes into a PR?
Happy to leave the more iffy stuff to a separate PR.

I got C++17 working for emscripten in this PR
https://github.com/openframeworks/openFrameworks/pull/6844/files

Just needed an extra header include and the C++17 flag in the end.

@roymacdonald
Copy link
Member

@ofTheo sure! no prob.

@Jonathhhan can you please point me to the repo in which you are pushing the emscripten upgrades

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Feb 7, 2022

Hey @ofTheo and @roymacdonald, thats great news.

I tried the url stuff from ofImage, but without luck.
But urls do work if I replace audioPlayer.load(ofToDataPath(fileName) with audioPlayer.load(fileName).
With the first one I can load files from the filesystem. Same for the video player.
Maybe that helps.

And I removed loadUrl()...

And here is the branch: https://github.com/Jonathhhan/openFrameworks/tree/AudioWorklet

I also made the audio context and the fft global (would also be possible to have a seperate fft for every sound id)...
I hope I didnt break too much... ;)
But I have all the older steps here if I should revert something...
And of course, take what you think makes sense and if you have any questions I am happy to answer.
Best.

@Jonathhhan
Copy link
Contributor Author

Actually the sound player works with url and local file with this methods:

	bool load(const std::filesystem::path& fileName, bool stream = false);
	bool load(const std::string& fileName, bool stream = false);

Somehow it does not work for the video player...

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Feb 8, 2022

This works for the video player (but maybe not a nice solution):

bool ofxEmscriptenVideoPlayer::load(const std::string fileName){
	if (ofFile::doesFileExist(ofToDataPath(fileName))){
		html5video_player_load(id, ofToDataPath(fileName).c_str());
		return true;
	} else {
		html5video_player_load(id, fileName.c_str());
		return true;
	}
}

I updated the branch, video and audio player have the method above now. This way I dont have to change a lot.
You can remove or change it if it is not save.

@roymacdonald
Copy link
Member

Hi @Jonathhhan
This is the chunk of code you need in order to tell if it is an URL or not. I think there should be an OF function wrapping this up. It can be quite handy.

#include "uriparser/Uri.h"

	auto uriStr = fileName.string();
	UriUriA uri;
	UriParserStateA state;
	state.uri = &uri;

	if(uriParseUriA(&state, uriStr.c_str())!=URI_SUCCESS){
		const int bytesNeeded = 8 + 3 * strlen(uriStr.c_str()) + 1;
		std::vector<char> absUri(bytesNeeded);
	#ifdef TARGET_WIN32
		uriWindowsFilenameToUriStringA(uriStr.c_str(), absUri.data());
	#else
		uriUnixFilenameToUriStringA(uriStr.c_str(), absUri.data());
	#endif
		if(uriParseUriA(&state, absUri.data())!=URI_SUCCESS){
			ofLogError("ofImage") << "loadImage(): malformed uri when loading image from uri " << _fileName;
			uriFreeUriMembersA(&uri);
			return false;
		}
	}
	std::string scheme(uri.scheme.first, uri.scheme.afterLast);
	uriFreeUriMembersA(&uri);

	if(scheme == "http" || scheme == "https"){
                // LOAD FROM URL. 
		//return ofLoadImage(pix, ofLoadURL(_fileName.string()).data);

	}

	std::string fileName = ofToDataPath(_fileName, true);
//LOAD FROM FILESYSTEM

I will go through all the changes, updates and upgrades you have done and get back to you.
cheers

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Feb 8, 2022

@roymacdonald thanks a lot. And just to mention: The old html5audio_stream_create method is not implemented in my branch yet, but I guess it can be easily replaced...
Without replacing it the stream works only with audioWorklet (but everything else should work independently from that).

@Jonathhhan
Copy link
Contributor Author

Jonathhhan commented Feb 8, 2022

Last update for now. The video player works now with url check (but I had to add "blob" to if(scheme == "http" || scheme == "https" || scheme == "blob" )). And the audio player with those 2 methods (because the url check didnt work there somehow - because the file / url is send as a std::filesystem::path and not as a std::string):

	bool load(const std::filesystem::path& fileName, bool stream = false);
	bool load(const std::string& fileName, bool stream = false);

And here are some of the updated examples: https://github.com/Jonathhhan/ofEmscriptenExamples

@ofTheo

Just needed an extra header include and the C++17 flag in the end.

Thats nice. I can confirm that it works, just needed to add your changes from ofConstants.h, ofFileUtils.cpp and ofUtils.h...

@Jonathhhan
Copy link
Contributor Author

Here are some good news regarding the future of Emscripten and Audioworklets (at the end of the discussion): emscripten-core/emscripten#12502 (comment)

@Jonathhhan
Copy link
Contributor Author

Just found another inconsistency between ofVideoPlayer and ofEmscriptenVideoPlayer with getDuration().
With ofVideoPlayer I get values between 0 and 1, while with ofEmscriptenVideoPlayer I get the duration in seconds.

@roymacdonald
Copy link
Member

duration between 0 and 1 sounds like a bug? Although what you get is dependent on the platform you are using. On which platform did you see this?

@Jonathhhan
Copy link
Contributor Author

Its Ubuntu, and I am sorry, I meant getPosition() of course...

@ranjithshegde
Copy link

I was able to get it to compile. I mostly needed to recompile the core lib dependencies.

@roymacdonald I have the same errors as you. Could you please inform me what is it that you recompiled ?

@roymacdonald
Copy link
Member

I was able to get it to compile. I mostly needed to recompile the core lib dependencies.

@roymacdonald I have the same errors as you. Could you please inform me what is it that you recompiled ?

I really cant recall. But i think that I recompiled emscripten using apothecary, maybe i had to recompile some other things using apothecary as well. That is what would make sense to me

@ranjithshegde
Copy link

@roymacdonald Thanks for your response. Sadly I dont know what apothecary is. I checked the git repo and it seems like there are some scripts in there, specific to emscripten was just a method to install it in docker.

Could you please point me towards any resources to understand what that repo is? Or how I would use them?

@roymacdonald
Copy link
Member

@ranjithshegde ah sorry. Apothecary is an openframeworks tool for keeping track and updating the libraries it uses. It is in github.com/openframeworks/apothecary
Just download that repo into scripts/apothecary and run from there. What you need to do is to find the emscripten "formula", which is also inside the scripts folder and change the version to the latest one and run apothecary so it updates emscripten

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants