Rendering Bug (video) with Pepper Plugins that causes Nw.Js to completey freeze since 0.46.0+

[anesuc]

NWJS Version : 0.48.0
Operating System : Ubuntu

Expected behavior

I have been using mpv.js to enable video playback outside the limitations of web codecs. It worked fine up until recently when I decided to update to the latest version of Nw.Js. The expected behaviour being that the video playback should work as it did before haha.

Actual behavior

The video playback initially lags and then freezes a few moments later. I have to force close Nw.js at that point. So I manually tested every single version from the latest 0.48.0 until I figured out the issue starts from 0.46.0. I'm hoping this will help figure out what changed from that point on wards to potentially cause this. I initially reported this issue on the official mpv.js plugin GitHub here, you can read there for a more detailed explanation. I was under the assumption it might be the plugin that needs to be updated. However, I have managed to test it on the latest version on Electron and it works fine. It hasn't been updated since I last downloaded it, so this is definitely caused by some change in Nw.Js. I do understand this is getting a little bit off base, but this happens to be the only way to enable decent video playback in Nw.Js. At this point a massive project is stuck on this version of Nw.Js and for a lot of reasons you can understand how worrying that is haha.

How to reproduce

As mentioned a more detailed explanation in in the link above. Here is an extract from there:

Here is a Linux version zip.

To run use npm run nw (or just ./nw)

• To see the code: it's in index.html
• Pre-built mpv.js in mpv.js directory. You can replace that with whatever for testing
• The nw.js files in here are from Nw.js 0.46.0. The same applies all the way up to the latest version. I tasted practically every one of them until this point. To test 0.45.5 for example, just download it from the official website here select the sdk build and just replace all files in the zip with the ones from there. Make sure to replace the libffmpeg.so in the Nw.Js directory (lib) using gcc -Wl,--no-as-needed -shared -lavformat -o /path/to/libffmpeg.so (this is to prevent conflicts as mentioned in the mpv.js GitHub).

Thanks

[frank-dspeed]

pepper is deprecated at all you should switch to wasm

[anesuc]

@frank-dspeed Interesting. It's important to note that the official Docs say it's still supported. On that note, do you know of any WASM options that enable video playback outside the limits of web codecs?

[frank-dspeed]

@anesuc you can compile any video codec to wasm simply use the c code of them. a good example is the google sqoosh pwa it uses C image codecs to render images same applys for video.

[anesuc]

I've been having a look and so far I've seen ones where the video goes through a transcoding process and then is passed along for playback using more processing power (my fans go on etc) than it should. I'll keep looking as I'm not farmiliar with webassembly to create something like that from scratch (and have a lot of other things to handle as this is a small part of a larger project)

[frank-dspeed]

@anesuc you don't need to be familar with wasm in general you can take it as a compiler wasm is simply LLVM-BACKEND

[sirisian]

@anesuc Did you find a solution to this? I noticed that mpv.js still works fine on the latest electron (Chrome 85), but is now broken on nwjs (0.48.4 at least) and says the plugin is not supported with no other errors. (I also noticed the latest 2 mpv dll's on windows broke with mpv.js, but that's a different topic). I was using mpv.js primarily for HDMI-In and sometimes HLS and other things and could never find an alternative.

In the meantime I'm looking at trying an HDMI to HLS device as I have a few. (Stability was an issue years ago with them though so I never made the switch. Seems better now).

[anesuc]

@sirisian Nope, decided to literally have it launch another instance using the old Nwjs for the video playback. I figured sooner or later someone else is going to stumble on this issue considering how important it is and there will be a solution at one point. Since security doesn't matter for this part it's not too much of an issue. The unnecessary memory usage etc on the other hand though...

[sirisian]

I just updated everything in mpv.js and also included the latest mpv-dev just to be sure with nw.js 0.49.0 (with and without SDK) and that didn't resolve anything. Specifically I'm seeing "This plugin is not supported" in the <object> tag. I'll test more ideas tomorrow specifically with older versions.

@frank-dspeed I think you might be misunderstanding what mpv.js does. It's currently the only way to run HDMI-In to an <object> tag and render it in an application (at least for Windows). It also supports RTSP streams. Essentially it starts an mpv player instance and renders the frame data into the tag context. Web assembly has no mechanism to accomplish either of these tasks as PPAPI exposes an OpenGL ES API which is way outside the scope of Web Assembly.

As for a side comment it seems like HLS.js at 1080p@60Hz with an HDMI to HLS converter box (like 130 USD) actually works consistently now. I'm still performing burn-in testing to see if the hardware is reliable, but for future readers this might be a simple solution, especially if pepper is ever actually removed. For RTSP I'll probably dive into WebRTC solutions.

[frank-dspeed]

@sirisian if it is about OpenGL what is missing in WebGL at present?

[sirisian]

The ability capture HDMI input and decode and render it efficiently. If you have to transcode or do other operations on the device it's generally too expensive on cheap hardware. This applies to RTSP generally also. Since the browser can't consume the protocol you have run it through something. (Whether that's a WebRTC server or VLC or whatever). Doing this on a cheap device can be unreliable and for 1080p@30Hz is usually too demanding.

[frank-dspeed]

i still do not fully get that out of my view You Need only to supply a Capture driver for the device that your running on as soon you have that you can simply use HTML to use that. In Case of NW.JS this could be a simple node module written in rust and then use neo bindgens to create a .node module thats it.

[anesuc]

I think the issue here overall is that mpvjs in general allows you to play all kinds of media types, subtle types (etc) and incredible levels of control that would require a lot more work to implement manually. It's equivalent to having a VLC-like module, with the idea that a user can literally throw anything at it and it should work without having (us the developers) work on every possible scenario. If one is just working on video playback this would be fine to do, but in a situation where that functionality is a small part of much bigger projects, it becomes a bigger task than we would want it to be.

[sirisian]

@anesuc As a follow-up to this I was able to build the latest mpv.js to work with mpv. I documented the minimal changes here: Kagami/mpv.js#76 (comment)

I haven't tested on Ubuntu yet, so your problem might still persist. Might test on the latest versions. (Everything is fine with mpv.js and Windows for me on the latest mpv and nw.js 0.51.0 with that custom built mpv.js node file).