Creating a video client side

[abacaj]

Hello, I am trying to make use of the webcodecs API, specifically using the VideoTrackReader to pass frames to VideoEncoder to encode frames.

I have tried to take the encoded frames and generate a new Blob(encodedChunks, {type: 'video/webm;codecs=vp8'}) but the resulting webm file is not playable.

I took a look at some of the examples and they seem to stop at the encoding stage and not generating a container for the video. Any tips for this?

Thank you

[aboba]

@abacaj Be aware that the interfaces under development as part of MediaCapture Transform are going to replace VideoTrackReader. The Intent to Prototype notice is here. This is why the VideoTrackReader interface has been removed from the specification.

[abacaj]

@aboba thank you for that. Will migrate to MediaStreamTrackProcessor. Would like to know from anyone here / community if they have been able to use the vp8 or other encoder to generate a video on the client side.

[chcunningham]

Sadly, I don't think we have any demos for this yet. Chrome folks are planning a demo hackathon later this quarter - I think this is a good candidate for us to work on.

I have tried to take the encoded frames and generate a new Blob(encodedChunks, {type: 'video/webm;codecs=vp8'}) but the resulting webm file is not playable.

Can you share the code? Are you pushing the encoded chunks directly into a blob? You'll want to use a library that muxes the chunks into the webm format.

[abacaj]

Hi @chcunningham thanks for that info. Yes, I'm new to video processing so missed that part about muxing to the container format. I was directly trying to use the encoded vp8 chunks as a blob.

I might be able to do it with ffmpeg.js (the emscripten port) fully client side. Will let you know how that goes.

[abacaj]

I did try to encode the blob using ffmpeg but received an error for invalid frame size, not sure what else I would need to do with the encoded frames.

[fferr] Input #0, image2, from 'test.raw':
log.js:15 [fferr]   Duration: 00:00:00.04, start: 0.000000, bitrate: 52230 kb/s
log.js:15 [fferr]     Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 2560x1440, 25 tbr, 25 tbn, 25 tbc
log.js:15 [fferr] Stream mapping:
log.js:15 [fferr]   Stream #0:0 -> #0:0 (rawvideo (native) -> vp8 (libvpx))
log.js:15 [fferr] [rawvideo @ 0x1a35040] Invalid buffer size, packet size 261152 < expected frame_size 5529600

[chcunningham]

I don't immediately grok your ffmpeg output, but you might be excited to see this demo that creates a webm file using a muxer implemented in javascript.

https://w3c.github.io/webcodecs/samples/capture-to-file/capture-to-file.html

[chcunningham]

Closing since I think the above sample may answer the question. Feel free to re-open if new questions arise.