emscripten - regression from #7617 - video files are not found

[ofTheo]

I think we have a regression from this #7617
Not sure why but since #7617 ( cc @artificiel ) moving the html to a directory and not copying the data/ folder videos now don't load.

emrun bin/opencvExample 

Now listening at http://0.0.0.0:6931/
 ofxEmscriptenVideoPlayer::load 
127.0.0.1 - - [01/Sep/2023 10:04:41] code 404, message File not found: /Users/theo/Downloads/of_v0.12.0_osx_release/examples/computer_vision/opencvExample/bin/opencvExample/data/fingers.mp4
127.0.0.1 - - [01/Sep/2023 10:04:41] "GET /data/fingers.mp4 HTTP/1.1" 404 -

copying data/ into opencvExample/ fixes the issue.

@Jonathhhan as he has some insight into how file loading is working from index.data

but this might warrant a patch-release.

[Jonathhhan]

my (superficial) understanding is that bin/data dir gets processed into a .data file by the --preload-file option (which currently is a problem if there is no data directory present: #7593)

Maybe it works without --preload-file after the filesystem get synced (it is needed after loading a new file into the filesystem)?

			FS.syncfs(true, function (err) {
				assert(!err);
        		});	

Not sure why some data types like images seem to work without --preload-file and others like videos not.

[ofTheo]

@Jonathhhan thanks!

I think the issue is that from the javascript side of things which html5video is, it is not able to access the files as its not emscripten C++ code.

But ofDirectory sees the file lying in data/fingers.mov even though it is not actually there from an html or js perspective.

eg:

So I think we need some sort of layer between ofxEmscriptenVideoPlayer and library_html5video.js which can either make the data/ folder exist or have the js have access to the --preload-file data

EDIT:
See the using files section here:
https://emscripten.org/docs/getting_started/Tutorial.html

It pretty much describes the problem.

[ofTheo]

I tried something like this to sync and read the file but while the FS.syncfs gets called the FS.readFile contents don't

    html5video_player_load: function(player_id, src) {
        console.log('html5video_player_load ' + UTF8ToString(src));

        // Read the contents of a file from the Emscripten file system
        FS.syncfs(true, function (err) {
            assert(!err);
            console.log('FS.syncfs');

            // Now that syncfs is complete, you can safely read the file
            FS.readFile(UTF8ToString(src), function(err, fileContents) {
                if (err) {
                    console.error('Error reading file:', err);
                } else {
                    console.log('File loaded - yay!');

                    const blob = new Blob([fileContents], { type: 'video/mov' });
                    const videoSrc = URL.createObjectURL(blob);

                    VIDEO.player[player_id].src = videoSrc;
                    var texId = GL.getNewId(GL.textures);
                    var texture = GLctx.createTexture();
                    texture.name = texId;
                    GL.textures[texId] = texture;
                    GLctx.bindTexture(GLctx.TEXTURE_2D, texture);
                    GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_MAG_FILTER, GLctx.LINEAR);
                    GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_MIN_FILTER, GLctx.LINEAR);
                    GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_WRAP_S, GLctx.CLAMP_TO_EDGE);
                    GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_WRAP_T, GLctx.CLAMP_TO_EDGE);
                    VIDEO.player[player_id].textureId = texId;
                }
            });
        });

        console.log('html5video_player_load 3');
    },

[ofTheo]

so just to be clear, the regression here is that video would work, but you had to upload data/myVideo.mp4 into the folder with index.html and then the library_html5video.js was actually just accessing the file locally instead of from the index.data

[artificiel]

the goal of #7617 is to minimize the manipulations required to “publish” an emscripten project. the current proposal needs a simple copy. maybe there are alternatives, generating a sort of « dist ».

is the video file actually bundled in the js data file? (i am unfortunately not in a position to check things now). if we end up requiring plain /data ressources, we don’t want to duplicate assets, so they should be excluded from the data file (?).

is the data file required or real advantage to a plain directory? (it does make things more complicated to update vs traditional web server management)

[Jonathhhan]

This patch shows how to load external media files, the filesystem is only needed for images: https://github.com/Jonathhhan/ofEmscriptenExamples/blob/main/emscriptenImport/src/ofApp.cpp

Other than that --preload-file seems the way to go, if data is in a folder different than index.html (but then there seems to be a problem, if the folder is empty or does not exist):
https://emscripten.org/docs/porting/files/packaging_files.html

[artificiel]

i had a moment to look at things; in short: packaging stuff into a dir reveals a design problem.

(TLDR for those unfamiliar with the emscripten data file like I was 20 minutes ago: it is an image of a directory hierarchy packaged in a file, and that hierarchy functions transparently like the system drive in regards to native C++ file interaction)

so ofToDatapath() builds the paths and return data/fingers.mov, and data/indispensable.jpg.

• ofImage works in pure C++, and "natively" finds data/indispensable.jpg within the emscripten "disk image";

• ofxEmscriptenVideoPlayer passes data/fingers.mov to javascript, which tries to find an url $CWD/data/fingers.mov, and the movie is not there.

so we have a system where some assets are bundled and managed internally by emscripten's "C++" filesystem, and some managed by javascript as relative URLs on the web server... it was luck that both worked, because they shadowed each other.

i find that "surprising"... especially since the contents of whole bin/data is copied into the data file (but for movies, they serve no purpose there).

i presume ofxEmscriptenVideoPlayer exists because the native C++ movie players don't work well in emscripten.

@Jonathhhan wrote :

the filesystem is only needed for images

my take is that it would make more sense to make ofImages work with a similar JS backflip to get back a full web server URL to pass to ofImage::loadURL(). (except when using emrun, the stuff will have to be downloaded one way or another)

it would also make it more friendly to asset management — as of now if you want to change an image in a deployed emscripten project you need the emsdk to re-generate/re-upload the data filesystem.

then, if we want to keep the idea of bundling an app into a meaningfully-named directory, it means data should be moved inside the directory, so the web server hierarchy stays logical with ofToDataPath().

but considering the compatibility with other platforms, better something like make dist that would create an ofApp directory, copy the contents of the the bin directory, and change ofApp/ofApp.html to ofApp/index.html. that would be the least intrusive vs other platform's behaviour (stuff works in command line with emrun, and when ready, make dist && upload.sh).

[Jonathhhan]

the filesystem is only needed for images

To be more clear:
The filesystem is (theoretically) not needed for ofVideoPlayer and ofSoundPlayer, because they use the html5 video / audio player which use urls and blobs for playback.

[ofTheo]

Yes, I should add to @Jonathhhan's reply.

The wasm filesystem is needed for everything except video/audio:
Fonts, images, 3Dmodels, xml, shaders etc.

I am not sure if uploading the data/ folder instead of packaging the files would end up causing more issues than we'd solve. ( I am not sure the C++ code can read from the web directory for example ).

i presume ofxEmscriptenVideoPlayer exists because the native C++ movie players don't work well in emscripten.

Yes, I think that is the case but it might be worth investigating some different approaches. I guess we could compile ffmpeg for emscripten if needed, but I think it will be possible to get the ofxEmscriptenVideoPlayer working. It's clear from the emscripten docs that it is possible and I think @Jonathhhan's code above provides some clues.

The very simple but not great solution though would be to exclude all video / sound files from being packaged and to copy just those files over to myAppFolder/data/

[ofTheo]

whoa - got it working...

hardcoded right now with the file name, but it seems to work:

 html5video_player_load: function(player_id, src) {
        console.log('html5video_player_load ' + UTF8ToString(src));
        
        
          try {
            var filePath = '/data/fingers.mp4'; // The path to your file in MEMFS
            var data = FS.readFile(filePath, { encoding: 'binary' });
            
            var stats = FS.stat("/data/fingers.mp4")
            var fileSizeInBytes = stats.size;
            
            const blob = new Blob([data], { type: 'video/mp4' });
            const videoSrc = URL.createObjectURL(blob);

            VIDEO.player[player_id].src = videoSrc;
            var texId = GL.getNewId(GL.textures);
            var texture = GLctx.createTexture();
            texture.name = texId;
            GL.textures[texId] = texture;
            GLctx.bindTexture(GLctx.TEXTURE_2D, texture);
            GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_MAG_FILTER, GLctx.LINEAR);
            GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_MIN_FILTER, GLctx.LINEAR);
            GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_WRAP_S, GLctx.CLAMP_TO_EDGE);
            GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_WRAP_T, GLctx.CLAMP_TO_EDGE);
            VIDEO.player[player_id].textureId = texId;


            // Handle the file content
            console.log('File size:' + fileSizeInBytes);
            console.log('data size:' + data.length);
          } catch (error) {
            console.error('Error reading file:', error);
          }

[ofTheo]

Okay, so I think the solution will work.
Will clean it up and submit a PR for both audio and video based on the above. Just need to account for when src is an http video :)

[artificiel]

indeed a partial data file (no video/audio) makes sense in terms of disk space but it's still quite strange (at least to me, a "newcomer" to emscripten app distribution) that some of my media will live as plain, accessible files on the web server, and some will need an emsdk preprocess packaging step. when I made the PR to bundle things in a directory I was under the impression that the data.fs was required and sufficient (an inference made because it's size matched the data dir, plus my test with (unfortunately) an image asset).

I guess I don't know if the goal of emscripten OF to (1) transparently "port" desktop apps with as little as possible changes to the src (using the same examples, for example), or (2) be more like iOS/android where it is quite implicit that an app designed natively there will not work on desktop, but will pull as much benefit as possible from the underlying platform.

1. transparent port: splitting the assets between plain and data.fs becomes an implementation detail that the user (me) should not care about (there is just "stuff" on the web server), but the product is not very web-ish as updating an asset requires an emsdk processing step, and dynamic action with the data dir is not possible with audio and video. but the local data dir stays integral for other platform compiles (and the data.fs only duplicates non-media files, which is a bit clunky, but probably OK).

2. more native experience: I would expect all assets to live in a data dir on the web server and I can manage them simply like I manage a data dir on a desktop app. but that will need all file-based things (shaders, etc as you mention) to support a loadURL method, and a form of server-side helper that allows a more intricate interaction with the ressources there (listDir() for example). and maybe there is a form of duality where things are searched in the data.fs, and if not found, tried with the shadowed URL. then users can tailor their use of data.fs, with the limitation that it does not support audio or video, unless an efficient audio/video player can be made in C++ (but I can imagine it would be an ordeal to keep up to date, while HTML5, once wrapped as it is, is basically "free")

so:

1. is definitely less work, and is probably required now and is achievable by rolling back Emscripten: wrap generated html into a directory #7617 plus filtering the contents of data.fs [[unless the HTML5 video player can read efficiently into a data.fs? perhaps possible with the FS module?]] and I still like the idea of a make dist that bundles things correctly for a simple upload;

2. seems like a future objective for someone who wants (or needs) to take a large bite...

[ofTheo]

@artificiel well the old approach before your PR was that files were getting bundled twice which definitely created more confusion and hid this issue.

I think having everything use index.data makes sense as a shorter term fix and I agree I am not sure if 2) would be a big project, but it could definitely be fairly involved.

In the end having the option for both might be handy - some people might want their files easily switched out, while others might want the security of having the files compiled to index.data.

[artificiel]

that’s super!

now that everything is bundled in data.fs it shifts a bit the problem which becomes being able to edit data.fs without being in the emsdk workflow. that could be offline (a kind of « image explorer » that mounts data.fs as a disk) or online (a form of server-side proxy to FS). i guess these 2 things already exist in some form somewhere…

but what’s nice about these 2 options is that they are completely external — from the OF/C++ perspective we have a clean and coherent solution.