WebRTC integration with camera RTSP stream sources

[allenporter]

Goal: Offer users the ability to use WebRTC with existing RTSP cameras to lower latency (near realtime)
Non-Goal: Not a replacement for HLS as the default streaming mechanism. We should still consider advertising LL HLS which offers significant latency improvements and still explore what it takes to make it enabled by default since it is a pretty nice alternative that requires no additional setup (it may be picky, but I think WebRTC may be even pickier. Even the http/2 requirement seems soft IMO)

Overview:
Prior proposal in #640 added WebRTC support for Nest, a cloud camera with native support. Following that proposal, this one also suggests we leave the complexity of RTSP outside of the core, but gives a well lit path for the signal path.

Proposal:

• Add a webrtc integration that is a very thin proxy to RTSPToWebRTC with a single method, something like async def handle_offer_for_rtsp(stream_source: str, offer: str) -> str that returns an answer.
• In camera:
  • In frontend_stream_type check if webrtc is loaded. If the stream_source starts with rtsp:// then return support STREAM_TYPE_WEB_RTC instead of STREAM_TYPE_HLS.
  • In camera.async_handle_web_rtc_offer provide a default implementation that invokes a single method e.g. webrc.handle_offer_for_rtsp. The current implementation sends a simple websocket request to the server.
• Package RTSPToWebRTC as an add-on. This integration takes an RTSP url and provides a WebRTC endpoint for it.
  The webrtc integration can be configured with a host:port of the RTSPToWebRTC server
• The webrtc integration supports auto discovery of the add-on similar to other add-ons like MQTT.
• We can leave the option open to swap out other server implementations in the webrtc component in the future. could consider other backends in the future.
• Integrations can still override frontend_stream_type and handle_offer_for_rtsp to provide their own implementations (e.g. nest or tuya in the future)

This is effectively providing a core version of the custom_component https://github.com/AlexxIT/WebRTC -- I have a proof of concept fo the above proposal working with an older nest RTSP camera using RTSPToWebRTC in a separate k8s container (I expect porting to an add-on will be straight forward)

Caveats & Future considerations:

• WebRTC is complicated (IMO). Following the nest proposal, this one also suggests we leave the complexity of RTSP outside of the core, but supports signal pass through
• This will likely put additional attention on the frontend webrtc code, which is very bare bones and likely needs a lot more work. I worry this may mean we're chasing lots of bugs.
• Users already configure RTSP cameras in home assistant, and we want to leverage that (e.g. not introduce additional CameraEntity or not introduce extra places for users to add camera configurations)
• Nest is a cloud camera, so NAT isn’t an issue. Supporting more RTSP feeds with a RTSPToWebRTC server may mean that users need ways to poke through NAT. Ideas here proposed in the previous discussion were that Nabu Casa might be interested in running a TURN server -- does the save on hls stream bandwidth?
  (However, as I think about this, it seems like if the RTSPToWebRTC server is the actual endpoint then it seems straight forward to poke a hole through the API, given the user was able to send an API request in the first place)
• Alternative: We could put this in the stream component, but it seems different enough to me that having it separate seems simpler.
• We could consider offering more configuration options per camera if there are too many caveats. Perhaps Pre-load Stream, LL HLS, and WebRTC could be modeled as per-device configuration options, though that might be difficult to port on to CameraEntity which isn’t a device.
• Additional work related to web rtc can be to consider integrations with alexa and google_assistant though really that is separate from this proposal.

This was sketched out with @hunterjm though correct me if I've missed any key points.

[hunterjm]

I think you covered it pretty well. I did realize after the fact that a TURN server is basically a proxy for the RTP stream and so wouldn’t save any additional bandwidth. That is why Google provides a free STUN server, but not a free TURN server.

I read somewhere on the interwebs that ~80% of negotiations can be resolved using just a STUN server to get the external IP, and WebRTC only uses TURN if it has to. I do not know how accurate that is.

Either way, using the existing video component for the frontend, we can default to WebRTC which would always work on internal networks, and fall back to stream if negotiation fails.

[allenporter]

@uvjustin curious if you have any input on this proposal for a separate WebRTC add-on based integration, and input into camera.

[allenporter]

Thanks, these are all great points.

I think we should chat more about:

• what it would take to make LL-HLS enabled by default
• documentation -- perhaps we can work through the "state of streams" discussion we started and come up with a plan to spell this all out with the community

@hunterjm had some suggestions around fallback from webrtc to hls on failures, which seems simpler and in line with what you're saying about TURN (either not worth the effort, or maybe just not needed at all).

[uvjustin]

Yes, we can discuss those other points too.
As for TURN, yes I don't see why that should be supported. In fact using a STUN server itself is kind of a connection failure, but it's one that NC can provide a helping hand with for little effort. TURN is really only required when all else fails, but in that case why even use WebRTC?
Here's another point worth thinking about: WebRTC (and HLS) are really supposed to be adaptive and provide multiple bitrates to deal with fluctuations in network conditions. Because we're not transcoding, we don't do this in HLS, and we presumably wouldn't be transcoding or doing this for the proposed WebRTC integration either. I'm not familiar with WebRTC internals, but I'd expect that WebRTC minus the ability to adjust bitrates would be less robust than HLS in dealing with network fluctuations, as the fragments/segments in HLS themselves provide a kind of internal buffer (a side benefit in exchange for the latency). On top of this, we've done work to get the H.265 codec (and its even better variations) playing in HA via HLS, but I'm not sure how easy it is to get H.265 working with WebRTC. H.265 streams require lower bandwidth than H.264 streams of the same quality, so this should give a H.265 stream more leeway with network conditions than an equivalent H.264 stream.
In other words, in trying to chase the extra 2 seconds of latency with WebRTC, we are probably going to be more susceptible to problems from imperfect network conditions.
@hunterjm Any thoughts on the above?
[Edited a few words and tagged Jason]

[allenporter]

These are all good points.

My initial thoughts are:

• Not using any additional servers sounds fine with me. We can keep it simple.
• We had a look at adaptive bitrate support in pion and it seems like a work in progress
• I did some auditing of the go binary, and its not bad.I sent a PR to fix a bug and got a pretty fast response. RTSPtoWebRTC appears like a demo project, though pion itself looks pretty well supported. I can see there are many caveats about what codecs are supported.
• I think the biggest improvement that can be made to the integration with rtsp to webrtc would be around making sure error messages are clear and easy to understand. If it just returns "Unknown error" it will be a pretty frustrating user experience
• Generally, it wasn't necessarily a goal of mine to have as few options as possible, but more to make it easier to use various options if desired and have them play nicely together. I'm still all for improving LL HLS and making it as a better default, though didn't necessarily see it as an either or.

[uvjustin]

Just a quick note - to achieve adaptive bitrates you need to control the encoder i.e. you are transcoding - that's what that discussion on the pion github was referring to. I think you need to transcode to really implement WebRTC, and IIRC that's one of the reasons @hunterjm shied away from using aiortc when developing stream.
I still think the key parts of WebRTC are 1) poking holes in firewalls to facilitate transport and 2) managing the media streams using adaptive bitrates. Since we wouldn't be doing either one of these I don't see a huge advantage in using the technology here (this is opposed to the great work you've already done for supporting WebRTC directly from devices that support it as that leverages both these features). It seems like WebRTC has a catchy name and has gathered traction among some fanboys in our user base, but I don't think they fully understand the technology.
OTOH, I don't think our implementation of LL-HLS is perfect either and it suffers from the same core issue of no transcoding. Additionally, there's more overhead with generating playlists, our control of the actual fragmenting process may not be fine enough as we outsource it to ffmpeg, and the transport layer may be too slow especially if using HTTP/1.1. An imperfect WebRTC implementation would still be an improvement in these areas. However, for me, the deal breaker with WebRTC is no H.265 support.

[uvjustin]

To clarify, I'm not against this per se, just setting expectations. I think a good use case for this is viewing within the home where real time viewing is desired and network conditions are robust. Also, a good extension of this might be 2 way calling for video doorbells.
Unfortunately I won't be able to help much even with testing as my setup is mostly H.265 and I'm remote from my HA server. However I think the implementation should be quite straightforward (you might have done most/all of it already).

[allenporter]

I think there are a few other considerations for webrtc besides adaptive bitrates: It's also using udp so it can drop packets w/o worrying about retry, and its usually in a push model w/o round trips for pull. That said, i haven't done lots of testing in poor network conditions, so i'm unsure. Perhaps it would be helpful to collect data, since I don't think we have it for any streams. (Given Streams are connected to a virtual camera entity, we don't exactly have device level diagnostics which is unfortunate. It could be interesting to create additional device level diagnostics for cameras.) ​

Yes, I have this almost already implemented.

I think the biggest takeaway is that this isn't a silver bullet, and we should be careful about how this is positioned. I'd like to think more about something like having cameras and streams be more declarative about supported codecs to get rid of this lowest common denominator worry we have with streams.

[uvjustin]

Yes, I was unclear and lumped the transmission of manifests + request round trip into the "generating playlists" part. LL-HLS is supposed to take care of the latency of the round trip with preload hints, but I'm not sure that all the players implement that properly. Also as we've seen, this can push the browser http connection limit. WebRTC is better on this front for sure.
As for dropping packets, it shouldn't be possible to arbitrarily drop packets in an encoded video stream since the frames are interdependent - dropped packets would cause video corruption until the next keyframe. I'm not familiar with its internals, but it's possible that WebRTC can detect when drops have corrupted the video stream and request a new keyframe, but there would still be video corruption before that keyframe is delivered.
Anyway, I'm waxing theoretical - if these are real issues they should have shown up in the custom component already. However, since there seems to be traction there, there's no harm in bringing that functionality in as a more official add-on.
Oh one more thing to note (this is not really related to whether we have it in HA or not) - enabling WebRTC in your browser can leak your IP even if you are using a VPN. This is probably not a concern for the users who want this feature, and users who haven't disabled the feature in their browsers might be leaking their IP already, but we should probably add a comment about this in future documentation just so users are aware.

[allenporter]

home-assistant/home-assistant.io#20306 is the initial integration, and wanted to share some quick thoughts so far.

The original name was webrtc but we decided that rtsptowebrtc is be more fitting given the specific service and specific nature of this integration. There may be other integrations like janus media service or camect or something that use webrtc in the future that would make webrtc be a confusing name. I tried to look at if there was commonality in how those could work, but it is not clear they would work the same because they have very thick javascript client libraries, and their own API endpoints like websocket endpoints built into the servers.

When thinking about how camera should integrate with this, I originally thought that camera should depend on this just like it does depend on stream and it can be very simple. However, it doesn't feel quite right to me giving the naming. We could instead extend camera with an exposed top level method like this:

def async_register_rtsp_to_webrtc_provider(def, provider: Callable[[str, str], [str])
    """Register a hook that can convert an SDP offer and RTSP stream source to an SDP answer."""

The integration can register itself with camera. Then camera can call the provider method to implement RTSP to WebRTC functionality discussed above. I was originally thinking about something more generic, or even trying to come up with a common implementation that also works for stream integration (e.g. RTSP stream source => Stream endpoint URL) thought it was not clear it was an improvement. Thanks @bdraco for talking through this idea.

[bdraco]

This is a good proposal, providing a path forward addressing the camera latency issue is key to providing a workable first class camera solution.

[allenporter]

This is a very good point, and has real practical issues to the lack of completeness of this approach. With this design, any rtsp to webrtc solution is external to home assistant and relies on that external process to do any transcoding (e.g. add on or external service). No recommendations to do this in core right now.

Specifically the RTSPtoWebRTC server does not support AAC and has an open issue with some ideas of what support for it would look like, but not actively moving.

My thought is it's a caveat a user can deal with. However, this is an example of why figuring out how to make LL HLS on by default or easier would be preferred today.