WIP / Discuss: Multiple video streams

[tcanabrava]

A tiny summary of what this does:

• Breaks a lot of code, do not think that I want to merge this, it's really a work in progress.

• [WARNING] Creates a new VideoSetting class that does not use the FactSystem (Thi is because I did not found out how use the fact system with multiple elements) - but code is sane and should be easy to handle. Give me pointers on how to convert this to FactSystem and I'll do.

• [WARNING] Breaks every single video control that we had. nor play, nor pause, nor record, nor pip, nor detach are working.

• Remove the old QtGstreamer implementation and uses a new GStreamer qmlsink plugin

• Uses gst_parse to create pipelines so we can remove a lot of boilerplate

• Handles 120 video streams (using a videotestsrc with a random pattern) (hard limited to 40 currently)

• Removes the Plugin for video streaming thru gtreamer: We don't need nor want this, the one thing we need is to let the developers define a pipeline that load their gstreamer plugins, we don't need to allow them to touch the manager or the video items to achieve this.

• Saves the VideoConfigurations for all the video in disk and restores them when opening the QGC.

Please, while looking on this review don't comment about the code that's broken, I know that a lot of things are broken. I want to know if this is a sane approach code-wise, to be merged in the near future (as soon as I fix the issue and clean the git history)

Thing that I still need to finish:

• Configuration on a per stream basis - it's there, I just need to plug the calls on Qml
• Rework the PIP for multiple video in mind.
• Rework Detach
• Rework Reattach
• Use a pipeline with udpsrc instead of videotestsrc

[DonLakeFlyer]

My two cents on this given limited understanding/experience with QGC+gstreamer:

• Current implementation has gotten messy over time as more and more feature have been piled on top of it
• Current implementation crashes a lot on Windows
• Assuming there is a new wayto do this more simply using the qmlsink plugin I think we should take a phased approach to reworking this code. This will make it way easier to code review smaller changes instead of one big massive change.
• First I would consider switching just simple video support to qmlsink. Pull all the bolted on feature out. Just make video streaming work.
• Verify that video streaming now no longer crashes on Windows. Sentera has user with laptops who can reproduce crashes 100% of the time.
• Get that change through.
• Now add the features back one at a time, one pull at a time.

@dogmaphobic Is the ultimate arbiter of all of this since he knows the most, but that is how I would approach this.

[tcanabrava]

@DonLakeFlyer This is easy to do with this series - I can cut the commits that add more features, and give you only the ones that are related to qmlglsink, while also working on the extended set on the side.

[DonLakeFlyer]

• Verify that video streaming now no longer crashes on Windows.

Is see this step as being critical. If we can't get the new stuff to not crash with no additional features added yet and just video streaming support then we should not proceed past this until it is figured out

[dogmaphobic]

Video sink and video receive/decode should be two separate things. For now, concentrating on getting a proper QML sink in place would be a huge value.

Video receiving/decoding needs a bit more thought as we need a better architecture in place. Using gst_parse is attractive but we need to be mindful of the resulting binary size. You would need to include the entirety of GStreamer, which on Android for example, means statically linking all of it, resulting in a binary twice its current size.

[andrewvoznytsa]

@tcanabrava, I tried to run your code to see how it works in overall. And if I run app built from tcanabrava:multiple_video_streams I'm getting

QQmlApplicationEngine failed to load component
qrc:/qml/MainRootWindow.qml:303 Type FlightDisplayView unavailable
qrc:/qml/QGroundControl/FlightDisplay/FlightDisplayView.qml:374 Type FlightDisplayViewVideo unavailable
qrc:/qml/QGroundControl/FlightDisplay/FlightDisplayViewVideo.qml:94 Type QGCVideoBackground unavailable
qrc:/qml/QGroundControl/FlightMap/QGCVideoBackground.qml:19 module "org.freedesktop.gstreamer.GLVideoItem" is not installed

Do you have idea what could be wrong? Perhaps something was not pushed to the repo?

I'm running that on Ubuntu 18.04/Qt 5.11.3/Gstreamer 1.14.5

[tcanabrava]

You need to install the gstreamer plugin that has the gl video item. It’s not in my code the error but a missing qml plugin. This code also uses a fake video profile, as don requested I’m finishing a version that handles just one video and is able to use a udp stream.

…

On Sun, 24 Nov 2019 at 14:53 Andrew Voznytsa ***@***.***> wrote: @tcanabrava <https://github.com/tcanabrava>, I tried to run your code to see how it works in overall. And if I run app built from tcanabrava:multiple_video_streams I'm getting QQmlApplicationEngine failed to load component qrc:/qml/MainRootWindow.qml:303 Type FlightDisplayView unavailable qrc:/qml/QGroundControl/FlightDisplay/FlightDisplayView.qml:374 Type FlightDisplayViewVideo unavailable qrc:/qml/QGroundControl/FlightDisplay/FlightDisplayViewVideo.qml:94 Type QGCVideoBackground unavailable qrc:/qml/QGroundControl/FlightMap/QGCVideoBackground.qml:19 module "org.freedesktop.gstreamer.GLVideoItem" is not installed Do you have idea what could be wrong? Perhaps something was not pushed to the repo? I'm running that on Ubuntu 18.04/Qt 5.11.3/Gstreamer 1.14.5 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8017?email_source=notifications&email_token=AA6XGUFCKXPOQ6VBK2ANOBDQVKBLRA5CNFSM4JL2XCE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFALYMA#issuecomment-557890608>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6XGUC3NZ2WMDDXOU5UMITQVKBLRANCNFSM4JL2XCEQ> .

[tcanabrava]

Showing the Multiple videos in action - I'm using gst-launch to generate a udp stream.

[Purvi-modi]

@tcanabrava What will have to be done to make the video controls working again?

[DonLakeFlyer]

Closing as abandoned

[mzahana]

Hi.

Are there any updates on multi video interfacing in QGC?
What is the current state, and what is missing ?

Thanks.

[tcanabrava]

It’s abandoned - if you would like to take over I can help you get started.

…

On Sat, 5 Dec 2020 at 11:47 Mohamed Abdelkader Zahana < ***@***.***> wrote: Hi. Are there any updates on multi video interfacing in QGC? What is the current state, and what is missing ? Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8017 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6XGUGDDMCTF75T5X77UFTSTIMTLANCNFSM4JL2XCEQ> .

[DonLakeFlyer]

I would do all of this through the new mavlink camera protocol which already supports auto-discovery of video streams and multi-vehicle video stream switching.

[mzahana]

@DonLakeFlyer can you please point me to the mavlink camera API/protocol?

[xdwgood]

@tcanabrava How to display multiple video streams simultaneously in the current qgc? Could you provide some help? Thank you