core: support audio streaming

[ES-Alexander]

Current behaviour

When connecting a microphone there's no option in BlueOS to stream the audio to the surface, or record it.

Expected or desired behaviour

Companion supported automatic streaming of the BR Low-Light Camera's audio. Would be good to support at least that, but preferably just audio streaming and recording in general (assuming that's possible).

Internal discussion here, which mostly covers:

• QGC currently doesn't have audio support, although there is an issue about it
• BlueOS builds gstreamer from source, and may not have the alsasrc element as a result
• audio support likely makes the most sense in mavlink-camera-manager (which BlueOS uses), but adding it would likely require some API changes, and may not be super simple to do
• definitely a useful feature to some users, but also not critical functionality to most users

Prerequisites

• I have checked to make sure that a similar request has not already been filed or fixed.

[ES-Alexander]

This issue has been raised in this forum post, which mentions:

what I want to do is connect a usb hydrophone and channel the audio to the ground station, and play it in parallel to the video, I know that Qgroundcontrol discontinued the audio, so I was planning to play the audio separately in VLC

[ES-Alexander]

@joaoantoniocardoso did you mention recently that you had some plans to work on audio support for mavlink camera manager? I have a vague recollection of that, but could also be misremembering.

[joaoantoniocardoso]

@joaoantoniocardoso did you mention recently that you had some plans to work on audio support for mavlink camera manager? I have a vague recollection of that, but could also be misremembering.

I actually don't have solid plans for it yet, but the data/stream part shouldn't be difficult, we first need to figure out what we really want with it, then we can see how it would be done.

Do we want to have only the camera's builtin audio muxed with the video stream, or do we want to allow any audio to mux into any video stream, like from a USB microphone? Or even, do we want completely separate streams with only audio? I would like to hear your thoughts about it :)

[ES-Alexander]

Do we want to have only the camera's builtin audio muxed with the video stream, or do we want to allow any audio to mux into any video stream, like from a USB microphone? Or even, do we want completely separate streams with only audio? I would like to hear your thoughts about it :)

@joaoantoniocardoso I forgot to respond to this before, but just came across it again.

Here are my thoughts:

• The primary requirement is for the BR camera's audio to be sendable to the topside
  • The Companion Software approach had a separate stream for it, so muxing is not a hard requirement
• A good solution should be able to handle audio inputs that don't come from video streams (e.g. from hydrophones and USB microphones)
  • Ideally it would be possible to mix in audio streams from the topside as well, e.g. for audio commentary, but that may be more in the scope of the topside software rather than needing to be handled by BlueOS
• From a recording standpoint I imagine (but don't know for sure) that it's simpler to record video and audio streams that are already muxed together
  • This is particularly relevant for camera+microphone combinations like the BR camera
  • If we support muxing, ideally it would be for arbitrary audio+video stream combinations, and ideally one video stream could be muxed with multiple audio streams (e.g. for left/right)
  • If muxing wouldn't automatically align the audio and video timing together correctly then I don't think it's worth doing, because the audio would then need to be adjusted relative to the video anyway
• Tee behaviour (like we have for video streams) is desirable, so that different systems can get access to the same audio
  • As an example, there may be separate programs for piloting and for providing a live-stream to an audience, that should both be able to access the same audio, and separate streams reduce the configuration and potential port conflicts at the topside (albeit potentially at the cost of some extra tether bandwidth usage)
  • This is a "nice to have", not a strict/blocking requirement - some audio soon is better than no audio for longer, it's just worth keeping in mind as "of interest"