Ditch OpenCV for FFmpeg?

[tux3]

Given the trouble we've had with the video code and OpenCV in particular, I'm considering replacing it with FFmpeg and rewriting the video code accordingly.
OpenCV is good, but it has some quirks that I'd gladly do without (resolution probing taking way too long, use of exceptions, strange crashes/freezes, etc).

Pros:

• FFmpeg is fast and has a lot of large users backing it
• OpenCV tends to crash in mysterious ways, FFmpeg less so
• We get fast image conversion for free and can directly interface with VPX/Qt.
• We could disable exceptions and get smaller/faster binaries
• Get rid of all the unneeded OpenCV image processing/GUI/math code we don't need.
• The capability APIs are non-portable, but they work. No more spending 10s probing the cam.

Cons:

• FFmpeg (the library) has barely any documentation. The best doc is reading existing code.
• Libav won't cut it, we need FFmpeg's libavdevice, this may complicate packaging.
• No cross-platform way to get/set device capabilities, one API per device (v4l2/dshow/qtkit/...)

So far I'm cautiously optimistic that this would be a substantial improvement in quality and stability, if I can show a reasonable way to work with device capabilities on all platforms.

[agilob]

👍 I had the same problem in my private project. I ended up removing opencv.

[tux3]

Initial testing with FFmpeg + Qt displaying the camera stream in the AV settings form shows really good performance.

I'll push a first draft in a new branch once I'm done fixing the crazy 10MB/s memory leaks from the first prototype.

[DanTheBritish]

Hopefully this would fix #829 on certain machines (eg mine)

[tux3]

I pushed a working FFmpeg version of qTox in a new branch. It should be ~stable, so feel free to test it.

However note that video calling between uTox and qTox on this branch currently shows wrong colors (inverted chroma planes). This can be trivially fixed in either client if needed.

[agilob]

I just tested your branch and:

1. Video has low number of FPS or is lagging, it's you know slow, I can see it's slow.
2. Camera preview is visible but combobox has still 'Initializing camera'
   
3. Each time I open aduio/video tab new 'Initializing camera' entry is added to the combobox (with resolutions).
4. Sliders are started at 0 and don't do anything.

[tux3]

@agilob

1. Video has low number of FPS or is lagging, it's you know slow, I can see it's slow.

Currently there's no resolution selection, so qTox picks up whatever the current default is. That probably happens to be the maximum your cam has, which would explain the lag.
We'll see if it still lags with a reasonable resolution, the code can definitely still be optimized. I made some compromises for more safety/stability at the cost of some performance.

1. Camera preview is visible but combobox has still 'Initializing camera'

Not Implemented™. Sorry, I should have mentioned that. The main focus was getting the base functionality up and running, and see if it turned out acceptable. The settings are coming.

1. Each time I open aduio/video tab new 'Initializing camera' entry is added to the combobox (with resolutions).

That's because I ripped out most of the old video code, and the new code for this combobox is not implemented, there's just this "Initializing camera" leftover.

1. Sliders are started at 0 and don't do anything.

Also not implemented, coming after I'm done with the resolutions combobox.

Thanks for testing!

[sudden6]

did you compare ffmpeg against gstreamer?

I think it has two advantages against ffmpeg:

• documentation available
• qt integration available

cons:

• I don't know about windows
• this info comes from just looking at websites, I did never do some actual coding

I don't want to say any of these is better, but each one is certainly more suited for the task than openCV.

[tux3]

I haven't looked in-depth at gstreamer, but at this point and after a cursory glance I haven't found any compelling advantage over FFmpeg.

documentation available

Getting started with little to no documentation was a pain, but once the core logic is up and running the FFmpeg doxygen and ffmpeg.c are enough to keep going. Of course it couldn't hurt to have more documentation, but it's not as much of a problem anymore.

qt integration available

That's really not needed, that's actually something I'd be wary of. All the Qt integration we need is provided by QImage being able to interpret and draw an arbitrary external RGB24 buffer without crazy overhead or copying.
What would be nice is some libvpx integration, we currently do some manual conversion between FFmpeg's AVFrame and libvpx's vpx_image (which isn't supposed to take an external buffer at all apparently).

It'd be interesting to look at gstreamer in more detail, but so far I think that FFmpeg is proving to be a good solution so far. The fact that FFmpeg has widespread use outside of GNOME is also motivating.

[sudden6]

ok on the documentation thing you are probably right.

But GStreamer isn't some gnome only thing, even kde bases his multimedia stuff on gstreamer, so it's probably even more used than ffmpeg.

[tux3]

But GStreamer isn't some gnome only thing, even kde bases his multimedia stuff on gstreamer, so it's probably even more used than ffmpeg.

On that point from wikipedia I'm seeing that FFmpeg is used in "VLC media player, xine, HandBrake, Plex, Blender, YouTube, and MPC-HC; it handles video and audio playback in Google Chrome", and while there probably is some use of gstreamer outside of GNOME, I haven't found much of it.

But in the end it's not all the relevant, the situation is mostly that I tried FFmpeg first and it has worked well so far, except for the ****AWFUL**** documentation.

[msteinbeck]

what does utox use?

[tux3]

Afaik uTox uses platform APIs directly (dshow, v4l2, ...)

We'll also do some of that, but only for the little things (listing the devices, resolutions, framerate) and not the actual video capturing/processing/conversion.

[tux3]

I added the FFmpeg code for Windows (DirectShow), everything should work except the hue/sat/lum sliders, those are still not implemented anywhere.
There's an experimental win64 qTox FFmpeg build here

This is also going to save some space, my current FFmpeg "lite" build is 4.3 MB upx'd, while OpenCV is 10 MB upx'd. And I can still make it much smaller by cherry-picking only the codecs we need for webcams.

[tux3]

Done.