[Poll] libvlc binding for Node.js with ability render video to html canvas. [$100]

[RSATom]

HI, I have idea for one else open source project, but don't sure if it will be useful to someone.

The Idea is simple - write node.js addon which could render video directly to html canvas via WebGL. I think this task is pretty feasible, but don't want spent time on it without be sure it will be useful and will be used.
So vote please if you like this idea.
Thanks in advance.

Did you help close this issue? Go claim the $100 bounty on Bountysource.

[smolleyes]

+1 ;)

Le 25/03/2015 08:17, Sergey Radionov a écrit :

HI, I have idea for one else open source project, but don't sure if it
will be useful to someone.

The Idea is simple - write node.js addon which could render video
directly to html canvas via WebGL. I think this task is pretty
feasible, but don't want spent time without be sure it will be useful.
So vote please if you like this idea.
Thanks in advance.

—
Reply to this email directly or view it on GitHub
#3293.

[shamun]

Also, please use the Gstreamer binding (in many many cases if VLC fails, then Gstreamer does the back up. i had lot of experience where VLC failed and Gstreamer extreamly helped like rescue)

NEW Stable Bindings: http://gstreamer.freedesktop.org/data/pkg/
(OLD get started reference http://docs.gstreamer.com/display/GstSDK/Installing+the+SDK)

[RSATom]

If i implement it for any one engine, adding of another one will be very trivial task...

[smolleyes]

hummm yeah but don t know for now but 2/3 years ago it was a BIG PAIN
to have gstreamer working on a mac :/ (but it rocks when it s working)

Le 25/03/2015 10:53, Sergey Radionov a écrit :

If i implement it for any one engine, adding of another one will be
very trivial task...

—
Reply to this email directly or view it on GitHub
#3293 (comment).

[RSATom]

It's another question. Anyway first of all I have to implement proof of concept, maybe even without libvlc or gstreamer at all.

[smolleyes]

good luck jaruba ;p

Le 25/03/2015 11:11, Sergey Radionov a écrit :

It's another question. Anyway first of all I have to implement proof
of concept, maybe even without libvlc or gstreamer at all.

—
Reply to this email directly or view it on GitHub
#3293 (comment).

[RSATom]

jaruba? :)

[smolleyes]

small mistake, sorry :) good luck anyway !

Le 25/03/2015 11:15, Sergey Radionov a écrit :

jaruba? :)

—
Reply to this email directly or view it on GitHub
#3293 (comment).

[shamun]

a) NOW: Maybe the good decision would be first to go with libVLC only, and try to make it stable, compatible at-least for Windows OS. Because most of the enterprises are using Windows OS.
(in OSX, Linux platform we have many other tools to make it work manually ourselves)

b) LATER: In case libVLC giving trouble or not expected quality then decide to use Gstreamer

Keep in mind, based on quality experience "Gstreamer" and VLC is good equal. But when compare them then you see one has good or less more better options, which helps in project requirements.

But i would go with the roadmap of libVLC as it will be stable for multi platform conceptually.

[mscreenie]

I'd like this for reasons of video compatibility but if I understand this correctly wouldn't an addon be the same as a plug-in? Whats the difference?

If writing this directly into nwjs, I would not be in support, it will add overhead for a feature that could be under used or used by only a few, plus nwjs is large enough as it is.

A better solution would be to use the new np-api's and release a plugin. Forgive me if I lack understanding on why this wouldn't be a better option or why NPAPI can not be used for high performance imaging applications.

There are good solutions for a work around, the best solution would be to stick with W3 compliance, the closest thing it would seem is plugins.

[RSATom]

NPAPI based plugin using libvlc internally already exists: https://github.com/RSATom/WebChimera
the problem is NPAPI support will be removed soon from Chrome/Chromium and as a consequence from NW.js

addon = C/C++ based plugin.

[mscreenie]

Thanks for the information, read up on NPAPI deprecation and a shame to see npapi dropped soon but maybe the right direction for web standards compliance and perhaps future innovation. Are you aware of any plans for Chrome or W3 to support support RTSP or any other live steaming protocols?

[RSATom]

not hear about it.

[jaruba]

+1, I'll port WebChimera Player to HTML, CSS and JavaScript if/when you do this

[arenabg]

+1 - will be difficult task, but i think that it will be useful

[RSATom]

thanks guys, already start working on it!

But poll is still active, so please vote if you like this idea.

[matthiasg]

@RSATom how exactly are you going to do this ?

We are investigating developing a virtual wdm capture device (that would at least help on windows machines, not sure what the equivalent on mac/linux would be). that way we could get any kind of memory access to a 2d surface within the browser via WebRTC.

[RSATom]

@matthiasg, I will write C++ addon for Node.js which will use libvlc internally to decode video to raw frames (to memory). After that I will draw that frames to HTML Canvas with WebGL shaders (I already do almost the same in WebChimera Plugin).
Nothing fancy.

[RSATom]

Idea with virtual cam is good, but don't solve problem with video frames transcoding to supported by video tag format.
But if you will just pass raw (decoded) frames via WebRTC, and draw them with WebGL - it could work. The only problem will be amount of passed data. Raw video frame is very big...

[matthiasg]

@RSATom in my case its not about video, its about interactive image manipulation. Basically in javascript you change a parameter (e.g position in 3d space) and an external renderer renders a new frame which needs to be transported to the browser.

I understand what you mean now btw with the c++ addon. I was thinking about a html only solution, while you are thinking about using the node.js part to somehow inject the frame into the html context. That will work of course and would be a less invasive method. I do like it, even though i would like it even more if either NPAPI were to remain or HTML5/JS would offer some kind of high-speed IPC.

[RSATom]

Unfortunately my solution will work only with Node-Webkit/NW.js, but not with Chrome...

[santeriv]

+1, just awesome. Any updates @RSATom ? how is this progressing?

[RSATom]

Working on this. Made some basic proof-of-concept, but met performance issues. Trying to find workaround.

[shamun]

Is it working yet @RSATom ?

[RSATom]

unfortunately I don't have too much time to spend on this, so it could take a while, but I'm still working on it. I hope I will have some solution before full NPAPI deprecation.

[Ivshti]

If this can work with reasonable performance it would be amazing. Obviously +1

[Ivshti]

This looks very badass.

As soon as you publish a reasonably working prototype, I will start working on a HTML5

[RSATom]

I hope I will have something workable (with linked libvlc) within two weeks (maybe less, just don't want promise), since now I have all necessary parts.

[Ivshti]

@RSATom this would work with atom/electron, correct? I don't see any reasons why it won't. Electron is literarily a few times more performant (especially in OS X - both in video playback and doing a lot of tcp work through node.js net), I don't know for what reason, I'm guessing the NW.js guys would figure it out.

[RSATom]

@Ivshti, If there will be way to get raw pointer (void*) from Uint8Array - it will work there too.

[Ivshti]

It looks like there will be no issue. Can I use your test code (cpp and js) to test it?

@zcbenz what do you think about this?

[RSATom]

@Ivshti, of course, but it's ugly and not works well.

[Ivshti]

@RSATom just tested it under electron, works fine. Did an updateFrame function called from JS just to see how the frame would update. Also requires re-initing the texture with gl.texImage2D

https://gist.github.com/Ivshti/a4900dfd3ea553a7a525
https://gist.github.com/Ivshti/b54fee57c58746d882b1

This results in a small 400x400 texture filled with black-white noise, and only consumes 4-5% CPU of one core in Electron. Pretty amazing.

I think using WebGL + node.js we can have an implementation that doesn't need to lock/unlock frames and still uses the same buffer - once texImage2D is called the frame is locked with the given buffers even though they are changing without re-allocation.

Allocating a buffer for every frame should not happen if not absolutely necessary IMHO, not because of the allocation overhead (as you mentioned in the VLC forum, this is not significant on desktop systems), but because of the deallocation GC overhead.

[RSATom]

I already did similar test but with linked libvlc and real video - it worked well. So we could treat concept as confirmed. Now I need reimplement it in less ugly manner.

I think using WebGL + node.js we can have an implementation that doesn't need to lock/unlock...

libvlc writes data in one of worker thread, so I anyway can't create Uint8Array there, and have to do some workaround. The problem is I don't know how big video frame will be, so can create it only when playing started - but I will get notify about it in separate thread, where I can't create Uint8Array. Anyway I'm already have one idea how workaround it.

[Ivshti]

That's great news. Can't wait to see it working :)

[RSATom]

Proof of concept is ready: https://github.com/RSATom/WebChimera.js
Build requirements: https://github.com/RSATom/WebChimera.js/wiki
Ugly demo: https://github.com/RSATom/WebChimera.js/tree/master/demos/ugly - you will have to change this line to something valid to try it out.

[RSATom]

This implementation is very basic and require some optimizations, but it works pretty well.

[jaruba]

I'm getting:

Uncaught Error: Cannot find module '../../build/release/WebChimera.js.node'

I guess I also need to build the .cpp files to get WebChimera.js.node for the demo?

[RSATom]

Yes, I'll write instruction how to build it soon.

[RSATom]

Binaries available: https://github.com/RSATom/WebChimera.js/releases/tag/proof-of-concept
Wiki updated: https://github.com/RSATom/WebChimera.js/wiki

P.S.: Please keep in mind, API is very limited, only play(mrl) and stop() now are available.

[Ivshti]

Great news!

Although I couldn-t build it - on OS X, I am getting those errors

../deps/libvlc_wrapper/vlc_audio.cpp:35:9: error: expected expression
        [event] ( audio_events_callback* callback )

for every expression of this style ([something]). I don't know what it does and I didn't have an idea what to do.

As for windows, I tried with the pre-compiled binary and the same nwjs build you listed (0.12.1 ia32) but it's giving me a "module not found" even though the file is in the specified path. Maybe it cannot link to the VLC dll?

[jaruba]

As for windows, I tried with the pre-compiled binary and the same nwjs build you listed (0.12.1 ia32) but it's giving me a "module not found" even though the file is in the specified path. Maybe it cannot link to the VLC dll?

-> RSATom/WebChimera.js#1

[RSATom]

I didn't try build it on mac, but I think it's lacks c++11 support

[mscreenie]

I'm working on a project that needs to display media from several sources including live video. This would be ideal.

But I was thinking and searching around for another solution by utilising native html5 and perhaps getting node to open a rtsp video stream and transcode into mpeg dash and run a local http server to be opened inside the browser.

It should work in theory and I'm planning to modify a project that already done this partly.

https://github.com/barbibulle/hippo/blob/master/server/node/README.md

I've done a few tests and it works well with local files. I accept there will probably be a delay of about 5 seconds but that's acceptable for my project.

[jaruba]

and perhaps getting node to open a rtsp video stream and transcode into mpeg dash

@mscreenie Doesn't transcoding use an immense amount of resources? I know it is far from being a viable choice in most projects that need extensive video type support.

[RSATom]

Good news guys, I've almost done with tasks scheduled for the first beta release. I hope it will out next week. Another good thing - it seems current implementation could be used not only with nw.js, but with io.js too (for example if you will need decoded frames without any visualization). And @Ivshti working on compatibility with Atom shell/Electron.

Current available API: https://github.com/RSATom/WebChimera.js/wiki/JS-API

[RSATom]

Checked compatibility with linux - it works! @Ivshti did the same for Mac OS X - it works too. It means we almost ready for first release!

[RSATom]

First public release is ready:
npm package: https://www.npmjs.com/package/webchimera.js
API: https://github.com/RSATom/WebChimera.js/wiki/JS-API
Usage example: https://github.com/RSATom/wcjs-ugly-demo

Prebuilt binaries will be available via npm a little bit later.

[jaruba]

WebChimera.js Player published at:
https://github.com/jaruba/wcjs-player

[JoshTheDerf]

Excellent! Thanks a lot for your amazing work! (Both RSATom for your insane skills and dedication and jaruba for writing an easy-to-use player so quickly)

[jaruba]

@Ivshti also deserves credit on this project, he helped a lot :)

[JoshTheDerf]

And thank you too @lvshti. :)

[RSATom]

Now we have pretty stable implementation, so we could close this issue.

[jaruba]

Now available on NPM:

• wcjs-player
• wcjs-renderer
• webchimera.js

[Ivshti]

Let me fill out in on what each module does.

WebChimera.js - libvlc binding, the JS api allows you to control libvlc and get raw frames in UInt8Array
wcjs-renderer - uses webchimera.js to render a video ( by drawing the frames in webgl )
wcjs-player - full featured video player based on the former two