generate webm from frames

[1N50MN14]

First thanks for the awesome work!

I'm trying to merge multiple frames (about 120) at 30fps into a single video but for some reason I'm not getting any feedback from the worker.

I'm sending the following arguments:

  var memfs = [];
  var args = ['-framerate', '30', '-i'];

  for (var i =0;i<images.length;i++) {
    var n = 'img-'+pad(i, 3)+'.png' //just padding the image names
    memfs.push({name: n, data:images[i]}) //pushing the images to memfs with ArrayBuffer data

    args.push('-i'); //adding those to args
    args.push(n);
  }

  args = _.concat(args, ['-c:v', 'libvpx', '-pix_fmt', 'yuv420p', 'out.webm'])

A simple --version works just fine, wondering if I'm doing something wrong here? would very much appreciate any help.

[1N50MN14]

Apologies ffmpeg returns -i: No such file or directory (I'm running it within Cordova/browserify)

[Kagami]

Seems like -i in this line var args = ['-framerate', '30', '-i'] is superfluous.
Command for converting still images to video should be -framerate 30 -i img-%03d.png -c:v libvpx -pix_fmt yuv420p out.webm though, otherwise you will get video with only single (first) frame.
But I don't think you would be able to run this because there is no PNG decoder in current builds. I can include it to the next version of ffmpeg.js.

[1N50MN14]

The images are in memory as mountfs doesn't work in Cordova. Wouldn't I need to have the images on disk or would that work since the images are listed in memfs?

As for PNG format, what does ffmpeg.js support, JPEG? I can convert the images (toDataUrl) before passing them.

Thank you

[Kagami]

would that work since the images are listed in memfs?

MEMFS should always work.

as for PNG format, what does ffmpeg.js support, JPEG?

Currently there are no image decoders, only for video.

[1N50MN14]

Oh I see... would it be possible for you to point me towards how I can generate a build with image decoders (any)?

[1N50MN14]

@Kagami Apologies for asking again about this (it's very urgent for me), I've looked at Makefile and read the build docs of ffmpeg, couldn't find any reference on how to compile with specific image decoders I'm under the impression this is somehow build-it (?), yet when I pass the jpeg frames I get the error Invalid data found when processing input... would it be possible for you to point me to the relevant library that I need to add to Makefile? Thank you

[Kagami]

Should work with latest version, I added JPEG decoder. PNG currently doesn't work because of emscripten issue.

[1N50MN14]

@Kagami Wow!!! Thank you so much!! This part works now excepting that the app runs out of memory and terminates :( Going to play around see if I can append N frames at a time instead; thank you again

[1N50MN14]

@Kagami Ok I managed to work around this, the issue I'm facing is I'm still receiving Invalid data found when processing input, I tried to pass the image data as Unit8Array, ArrayBufferor even Blob... my setup is still the same (minus the additional -i):

  var memfs = [];
  var args = ['-framerate', '30'];  
  var len = images.length;
  for (var i =0;i<len;i++) {
    var n = 'img'+pad(i, 3)+'.jpg'
    memfs.push({name: n, data:images[i]})
    args.push('-i');
    args.push(n);
  }
  args = _.concat(args, ['-c:v', 'libvpx', '-pix_fmt', 'yuv420p', 'out.webm'])

And I'm using the worker version:

      worker.postMessage({
        type: "run",
        MEMFS: memfs,
        arguments: args
      });

Not sure what I'm missing on, I see you added jpeg decoding to the tests, could it be the worker version not updated or I'm incorrectly using memfs?

[Kagami]

Your command should be -framerate 30 -i img-%03d.jpg out.webm if you want to create video from image frames.

Try to start with simplest case, convert single JPEG image to WebM.

[1N50MN14]

The problem with -i img-%03d.png is that it's being treated as a single image input so I receive the error no such file or directory. Anyway I tried with a single image as you suggested like so:

var memfs = [{name:'img-001.png', data:images[0]}];
var args = ['-framerate', '30', '-i', 'img-001.png', '-c:v', 'libvpx', '-pix_fmt', 'yuv420p', 'out.webm'];

With the image data being Uint8Array, both pngand jpegvariations, still receiving img-001.jpg: Invalid data found when processing input...

[Kagami]

is that it's being treated as a single image input

It shouldn't, try that with standalone ffmpeg. Or maybe try -pattern_type glob (like in tests).

still receiving img-001.jpg: Invalid data found

Are you using latest version? Try different image. Also try it on node platform just in case.

[1N50MN14]

I have already tried pattern_type like in the tests, I then get an error Unrecognized option 'pattern_type'.. I'm using the latest version 3.1.9000, ffmpeg-worker-webm.js.

Now going to try ffmpeg-webm.js in both node and browser

[1N50MN14]

Ok this is weird... using ffmpeg-webm.js in the browser it doesn't complain about pattern_type but I'm getting an input alert window (!) with input:label and Cancel / Ok buttons ;)

var memfs = [{name:'img-001.jpg', data:images[0]}];
  var args = ['-pattern_type', 'glob', '-framerate', '30', '-i', 'img-001.jpg', '-c:v', 'libvpx', '-pix_fmt', 'yuv420p', 'out.webm'];

  var result = ffmpeg({
    MEMFS: memfs,
    arguments: args,
    print: function(data) { stdout += data + "\n"; },
    printErr: function(data) { stderr += data + "\n"; },
    onExit: function(code) {
     console.log("Process exited with code " + code);
     console.log(stderr)
     console.log(stdout);
    }
  });

[Kagami]

Maybe latest Emscripten brought some breaking changes in case of browser environment. Will check that.

[1N50MN14]

@Kagami Yeah... sorry about the hassle...

[Kagami]

It was a stdin request… Probably worth to mute it by default in order to avoid confusion, it should be barely needed. Here is full browser example, works for me just fine:

<script>var module = {};</script>
<script src="node_modules/ffmpeg.js/ffmpeg-webm.js"></script>
<script>var ffmpeg = module.exports;</script>
<script src="index.js"></script>
<video id="video" autoplay loop></video>
<br><a id="link" download>download</a>

function fetchData(url) {
  return new Promise(function(resolve, reject) {
    var xhr = new XMLHttpRequest();
    xhr.open("GET", url, true);
    xhr.responseType = "arraybuffer";
    xhr.onload = function() {
      if (this.status >= 200 && this.status < 400) {
        resolve(new Uint8Array(this.response));
      } else {
        reject(new Error(this.responseText));
      }
    };
    xhr.onerror = reject;
    xhr.send();
  });
}

Promise.all([
  fetchData("/1.jpg"),
  fetchData("/2.jpg"),
]).then(blobs => {
  var res = ffmpeg({
    MEMFS: blobs.map((data, i) => ({name: `${i+1}.jpg`, data})),
    stdin: function() {},
    arguments: [
      "-framerate", "1",
      "-i", "%d.jpg",
      "out.webm",
    ],
  });
  var videoBlob = new Blob([res.MEMFS[0].data]);
  var videoUrl = URL.createObjectURL(videoBlob);
  document.getElementById("video").src = videoUrl;
  document.getElementById("link").href = videoUrl;
});

It's very basic and a bit hacky, but you got the idea.

[1N50MN14]

oh I see, thank you very much I'll check it right away

[1N50MN14]

It worked!!!! THANK YOU!!!!! Just one minor thing which I can't figure out is how to get the result with the worker version, it seems that msg.type doneis not being called, I'm assuming the uint8array would be in msg.data?

  worker.onmessage = function(e)
  {
    var msg = e.data;
    switch (msg.type) {
    case "ready":
      console.log('**** worker ready, merge')
      worker.postMessage({
        type: "run",
        MEMFS: memfs,
        arguments: args
      });
      break;
    case "stdout":
      console.log(msg.data)
      stdout += msg.data + "\n";
      break;
    case "stderr":
    console.log(msg.data)
      stderr += msg.data + "\n";
      break;
    case "done":
      var mem = msg.data.MEMFS;
      console.log('DONE!!', mem)
      worker.terminate();
      break;
    case "exit":
      console.log("Process exited with code " +msg.data);
      console.log(stdout);
      console.log(stderr)
      worker.terminate();
      break;
    }
  };

[Kagami]

Yea, resulting files should be in data field of done message.

[1N50MN14]

Wow!!! Works like a charm!!!! Thank you SO SO much for saving the day for me!! Please let me know if you ever swing by Sweden I owe you beer!

[tolstenko]

Thanks for you work! It is awesome! Please create a wiki page detailing these workarounds, and how to use arraybuffers from jpeg files to pass them to ffmpeg.
Please dont give up from compiling with png.