read a large file and stream to rtmp

[navinSing]

I am using

https://www.npmjs.com/package/read-blob

to read a large file from chunks ..
how can i use ffmpeg,js to stream these chunks to rtmp

[1Syler]

This link has detail for streaming segments https://blog.scottlogic.com/2020/11/23/ffmpeg-webassembly.html

[cakekindel]

@1Syler that link only seems to explain how to get ffmpeg to output in chunks, not how to get ffmpeg.run to read from a stream. My assumption is that MEMFS buffers the entire file in memory which could cause issues if we're dealing with (multiple) large files concurrently, right?

Edit 1

Reading more on Emscripten's filesystem API, found WORKERFS, which doesn't copy the entire file to memory. would it be possible to use this with ffmpeg.wasm?

Edit 2

If I'm not crazy, I think I've answered my own question by digging around this repo and core a bit on my own. Would love some feedback on this solution, but it seems roughly doable!

The only blocker is that the core script would have to be built with WORKERFS explicitly opted into. (good opportunity for a small fork?)

I couldn't come up with a way to write a JS stream to a file on the user's machine other than the experimental FileSystem API which restricts this example to new versions of Chrome and Edge.

Any & all feedback welcome.

High-level:

1. get a File object referring to a source video
   • This could be done with a "choose file" button, rather than FileSystem
2. get a writable stream from a file save dialog
   • No alternatives I could come up with
3. mount the File in emscripten's WORKERFS filesystem
4. transcode it (normally or chunkwise)
5. stream the output to the writable stream from 2

(near) full example

// main.js

const avi = {
  description: 'AVI Video File',
  accept: { 'video/x-msvideo': ['.avi'] },
};

const mp4 = {
  description: 'MP4 Video File',
  accept: { 'video/mp4': ['.mp4'] },
};

const sourceFile = () => window.showOpenFilePicker({ types: [avi] })
                               .then(entry => entry.getFile());

const destWritable = () => window.showSaveFilePicker({ types: [mp4] })
                                 .then(file => file.createWritable());

someButton.onclick = async () => {
  const sourceFile = await sourceFile();
  const destWritable = await destWritable();
  
  // spawn the child thread that will do the transcoding
  const childThread = new Worker('./ffworker.js');

  // do the stuff
  childThread.postMessage({sourceFile, destWritable});
  
  // wrap up
  childThread.onmessage = () => {
    destWritable.close();
    childThread.terminate();
  };
};

// ffworker.js
const fs = require('fs');
const { createFFmpeg, fetchFile } = require('@ffmpeg/ffmpeg');

const ffmpeg = createFFmpeg({ log: true });

// this will be invoked when `main.js` calls `childThread.postMessage({...})`
onmessage = async ({sourceFile, destWritable}) => {
  await ffmpeg.load();
  
  // mount a `WORKERFS` file system to a directory
  // and stick `sourceFile` into it
  ffmpeg.FS('mkdir', 'working');
  ffmpeg.FS(
    'mount',
    WORKERFS, // no clue how to import this, emscripten's docs are confusing
    { files: [sourceFile] },
    '/working'
  );

  // do some magic here to:
  // - transcode (normal or chunkwise)
  // - write every output chunk to `destWritable`
  // - when done do `postMessage()` to close `destWritable` and terminate the worker
};

[soamsy]

For one of my projects, I ended up creating my own fork of ffmpeg.wasm-core to get WORKERFS opted into, if anyone's interested. To enable WORKERFS, I had to add -lworkerfs.js to the flags in the ffmpeg core build script, and then add the FS_mount, FS_unmount, FS_filesystems methods to emscripten's EXTRA_EXPORTED_RUNTIME_METHODS so I could interact with it (FS_filesystems is the important one, since that's where WORKERFS is).
ffmpegwasm/ffmpeg.wasm-core@n4.3.1-wasm...animebook:n4.3.1-wasm

To actually use WORKERFS, you need to call the library from a web worker. This required me to use the ffmpeg core library directly, since the ffmpeg.wasm wrapper library actually errors out right now if you use it from a worker, mostly due to its use of the document object, which doesn't exist in a worker context. The document object's only used to resolve urls as far as I could tell, so I got by without it. This involved copy/pasting most of the ffmpeg.wasm code, which you can see here for full details: https://github.com/animebook/animebook.github.io/blob/d20cea4d4823df07f6f0f3c2056979cd3c1444e4/extension/bg/js/ffmpeg.js

If I had to update your ffworker.js example, it might look something like this with WORKERFS, and just the core library. Not a full working example, since there's more you'd need to do to wrap ffmpeg core, such as detecting when the ffmpeg command is done with log messages, but it should get the rough idea down.

// ffworker.js example
self.importScripts(
    'ffmpeg-core.js', // import ffmpeg core library
);

onmessage = async ({sourceFile, destWritable}) => {
  const ffmpegCore = await load();
  const FS = ffmpegCore.FS;
  const WORKERFS = ffmpegCore.FS_filesystems.WORKERFS;

  FS.mkdir('working');
  // mount a `WORKERFS` file system to a directory
  // and stick `sourceFile` into it
  ffmpegCore.FS_mount(WORKERFS, { files: [ sourceFile ]}, '/working');

  // do some magic here to:
  // - transcode (normal or chunkwise)
  // - write every output chunk to `destWritable`
  // - when done do `postMessage()` to close `destWritable` and terminate the worker
};

async function load() {
    // create ffmpeg core directly
    return await createFFmpegCore({
        mainScriptUrlOrBlob: '/full/path/to/ffmpeg-core.js',
        locateFile: (path, prefix) => {
            if (path.endsWith('ffmpeg-core.wasm')) {
                return '/full/path/to/ffmpeg-core.wasm';
            }
            if (path.endsWith('ffmpeg-core.worker.js')) {
                return '/full/path/to/ffmpeg-core.worker.js';
            }
            return prefix + path;
        },
        printErr: message => {}, // need to implement this to detect when ffmpeg's done, see how ffmpeg.wasm does it
        print: message => {} // need to implement this to detect when ffmpeg's done, see how ffmpeg.wasm does it
    });
}

Also, this is probably something I only ran into in my project due to how my chrome extension is architected, but I ran into some permission problems with sending the file to WORKERFS due to how I was sending a file to an iframe, which then sent the file to a worker. I don't think anyone other than me is doing that, but if you get permission problems for any reason, I got around it by recreating the file before sending it anywhere. It appears to be a chromium bug with sending File objects across different contexts. https://github.com/animebook/animebook.github.io/blob/d20cea4d4823df07f6f0f3c2056979cd3c1444e4/extension/fg/frontend.js#L49-L53

const cloneFile = new File( [ file.slice( 0, file.size ) ], file.name, { type: file.type } );

[samelie]

hello @soameshi Thanks for sharing your work.

Trying to compile locally with bash build-with-docker.sh but getting stuck on the vorbis step: libtoolize: error: linking '/usr/share/libtool/build-aux/ltmain.sh' to './' failed
Do you have any tricks to compile?

[soamsy]

I personally had trouble getting docker to work at the time, and since my changes were small I used the github actions already being used with ffmpeg.wasm-core to compile the project e.g. https://github.com/animebook/ffmpeg.wasm-core/actions/runs/763252454

Probably not a long-term solution, but if you want to just play around to see what WORKERFS can do it's an option. I haven't tried to replicate the ubuntu setup the github actions uses yet.

[samelie]

Thanks for the reply, good call on the github actions. And also thanks for sharing your project where you included the compiled WORKERFS wasm; a lot of really clever ideas in there.
btw, for others, crossposting [the issue I closed out] regarding the error(#182) - the problem was an older deprecated docker-for-desktop on mac osx high sierra.
-sam