Handling large files

[LucCADORET]

First of all, great job on the library.

I have been trying to find a library to convert .mkv to .mp4 (which is basically changing the container while keeping all the codecs most of the type, so pretty fast), but I am struggling with it. videoconverter.js, and ffmpeg.js (from @Kagami) are both failing at handling "large" files. My test file is 149. MiB, and if always end up with a memory overflow crash (in node and in browsers).

I have been doing the test with the same file with this new library, and I stumble upon "RangeErrors", again both in node and browser version. (JSON.stringify when sending the message to the child process seems to be the part where is fails, in node).

I think that the library should be able to handle larger files, as being able to convert video files is pretty much useless if you can't handle files with 'real world case' sizes.

[jeromewu]

Hi, in ffmpeg.js v0.4.1 we have fixed this bug and feel free to try with the CodePen below:

https://codepen.io/jeromewu/pen/NWWaMeY

[gsulloa]

Hi! I'm using ffmpeg.js v0.5.2 trying to encode a video of 276MB, and is not working. I've tryied in your codepen, but also fail with null error. Is there any workaround for this?

[caugner]

It looks like there is a limit for IndexedDB items:

• Chrome: ~133 MB (https://stackoverflow.com/questions/24362839/what-is-the-per-record-size-limit-of-indexeddb#comment92785955_24362839)
• Firefox: ~267 MB (https://bugzilla.mozilla.org/show_bug.cgi?id=1439704)

Somehow, the videoconverter.js does not have this limitation.

Update: This may be due to videoconverter's usage of emscripten's FS.createDataFile here:
https://github.com/bgrins/videoconverter.js/blob/42def8c4136b03cbe55951fbf6fcda214f51059a/build/ffmpeg_pre.js#L32

According to the docs, Worker.write "writes data (...) to Emscripten file system", but the browser implementation actually writes to the IDB.

[caugner]

@jeromewu I wonder whether WORKERFS [1] could be used, cf. example [2]?

[1] https://emscripten.org/docs/api_reference/Filesystem-API.html#workerfs
[2] https://emscripten.org/docs/api_reference/Filesystem-API.html#FS.mount

[LucCADORET]

Personally, I got around this issue by using ffmpeg core (used why this library), and passing the data to the worker using a Transferable in the worker.postMessage() method (a transferable basically tells passes the data by reference, and not by copy, which is way better when you're trying to pass a big amount of data).

And then in the worker, I think it's using the MEMFS.

I'm able to go up to 1.8GB it seems (though I didn't do accurate tests), you can check that out here http://comeover.io/home

[caugner]

@LucCADORET Thanks for the hint. I'm hoping for a fix in the worker provider to avoid the effort of maintaining my own worker. 😉