Support all video formats by falling back to Javascript codecs

[mcarroll76]

Following on from this closed issue: #9

This can all be done in the browser with javascript. Here are some examples of codecs written in JS:
https://github.com/phoboslab/jsmpeg
https://github.com/mbebenita/Broadway
https://github.com/audiocogs/aurora.js

Here's a video convertor in JS: https://github.com/bgrins/videoconverter.js

Here's FFmpeg, which works with most formats, compiled into JS: https://github.com/Kagami/ffmpeg.js

This was done with Emscripten, which can also compile into WebAssembly for those browsers that support it. Obviously, it's not going to be as efficient as the browser's native playback, but it's better than nothing.

So what we need is to modify the current player to check browser capabilities, play natively where possible, and fall back to WebAssembly then javascript players when that fails.

Someone's already created a player for Ogg formats here: https://github.com/brion/ogv.js/

If the same could be done with libavformat and libavcodec from FFmpeg, we'd have a player that could play nearly anything.

Some relevant discussion here: Kagami/ffmpeg.js#12

[Instagit]

Hey there,
for reference, we talked about video handling in the past here and here.

Decoding with JS is interesting, but is still in a very early state. Using it by default in enterprise quality software is not an option. Then there's also the general overhead from doing decoding with JS, which can cause performance problems for many devices. We always have to consider all use cases.

I see two ways to go about this:

1. Make an extension. That's how we ended up with this video player. It was originally an ownCloud extension. I put it in the app store, it was somewhat popular and got adopted as default player. You can fork the existing player if you want to. I would recommend the MediaElement.js branch which has support for SRT subs and more player functionality.
2. I could see JS decoding as an experimental setting if the core team agrees to it. Same as 1. it would still be good to have a working proof of concept.

In either case, I'd request some help as I'm just a contributor myself.

[mcarroll76]

I'd seen those previous discussions. Lots of talk about transcoding on the server, which would obviously be a horrible CPU hog, or using browser plugins, a deprecated technology. A Javascript decoder would be preferable to either of these options - it puts the load of video decoding in the browser where it belongs, and works in a standard browser.

As for performance issues, I'm looking at it from the perspective that something is better than nothing. It may use a lot of CPU on the client machine and maybe stutter if there isn't enough available, but currently these files simply don't play.

Writing codecs in Javascript from scratch would obviously a big job, but using Emscripten to compile the existing C code of libavcodec and libavformat to asm.js/WebAssembly might actually outperform hand-written code anyway. In recent years browser engines have been designed to optimise the limited subset of Javascript it compiles to.

The ogv.js player, which was created in this way, is already being used by Wikipedia as a polyfill to add Ogg support to Apple browsers. It's been adapted as a plugin to video.js here:

https://github.com/hartman/videojs-ogvjs

As the Nextcloud player is based off video.js, it should be straightforward to add Ogg support using that plugin. It would require more work to incorporate libavcodec and libavformat into the plugin, but the framework is there.

Unfortunately, my very basic Javascript skills are not at a level to do any of this myself. I just saw the previous discussions that didn't mention this potential solution, and I thought I should point it out.

[bvibber]

If anybody's interested in helping with making the libavcodec-based or other custom plugins, I'd be happy to do further work to separate the ogv.js player frontend from its backend and stabilize the plugin interface. (Being US-based I don't want to directly fiddle with the patented codecs too much myself, and we only need Ogg and WebM for Wikipedia's use case for now.)

[Instagit]

Lots of talk about transcoding on the server, which would obviously be a horrible CPU hog, or using browser plugins, a deprecated technology.

Agreed, that's why we left it as it was. There really is no perfect solution for handling videos in the browser right now.

I'm looking at it from the perspective that something is better than nothing. It may use a lot of CPU on the client machine and maybe stutter if there isn't enough available, but currently these files simply don't play.

As a user, I agree. As a developer who has to handle the tickets when something doesn't work as the users think it should, it's different. Many users won't understand why some videos cause stutter and others don't. Letting them know that the format is not supported is a more comprehensible way to handle this.

That's why I suggested turning this idea into an extension or an experimental feature. That way users who want it can have it and they'll understand that it may cause problems.

As the Nextcloud player is based off video.js

I spent some time improving the video player and the probable future version is based off MediaElement.js. It has more player features and supports SRT subs. See discussion here and branch here.

I just saw the previous discussions that didn't mention this potential solution, and I thought I should point it out.

Thanks for bringing it up. I'm definitely considering JS decoding, but like the other approaches, it comes with potential problems that need to be handled.

[Instagit]

I'm closing this for now, but it's not forgotten.