REST API for Stream Sources

[allenporter]

Summary

Expose a REST API in Home Assistant to make it a provider of camera stream sources to the ecosystem

Motivation

Imagine a user journey like:

• I've done the work to set up Home Assistant, including some cameras
• Now i'd like to go deeper! Say I want to use something like Frigate to do more fancy processing of my camera streams.
• I've installed Frigate as an add-on for Home Assistant, and it is able to query home assistant for my camera sources
• I open the Frigate and see my Home Assistant cameras recognized
• Now I can go deeper and configure additional options

In practice what's are the practical considerations to stop that from happening? The main roadblock is I need to teach Frigate about my cameras first.

• My local cameras expose an RTSP stream URL that is straight forward to configure. I can copy my rtsp stream url from home assistants config file to frigates config file. (e.g. for generic). That's not so bad.
• What about my cloud based cameras? Those may or may not have an easy to grab URL. Often a cloud provider would add additional security on a stream url e.g. may require additional url parameters with auth tokens.
  • As an example way users work around this today is to expose a public URL for their camera. Basically, users need to compromise on security/privacy features because they prefer usability to just get things working.
• What about X? There is a fairly long tail of camera integrations ( around ~40, though not sure how many support rtsp streams) different camera integrations so not sure how they're all configured... maybe some blend of the two extremes above

Proposal

Expose a REST API in camera. An example GET request:

/api/camera/stream_source/<entity_id>

With an example response:

{
    "stream_source": "rtsp://192.168.1.1/live",
}

Caveats & Alternatives

• Additional options
  • Maybe we can avoid options like stream_options used by generic and onvif. Generic is easy to configure directly in frigate. Onvif has rtsp_transport so if needed we could make this an explicit option to return. Generally, we would not want to return a generic dictionary of ffmpeg flags
  • Expiration: We could have an expires_at option. I'd like to see if we can avoid this, and just ask for updated stream sources when a failure happens.
• WebRTC would not be provided due to limited support everywhere at this time. However, it could work with a similar model: Pass in an answer and get back an offer.
• Specifically to Frigate: There may be other things to help make adoption easier, such as pulling into an official integration / add-on, but not discussing that here. (May also be that this is not the best idea given USB Accelerators are difficult to come by.)

We've had some initial discussions about this at various times over the past few months with @hunterjm @blakeblackshear @uvjustin but wanted to pull this into a real architecture discussion.

[allenporter]

Giving this discussion a poke for feedback.

The above proposes a Rest API for pulling streams, though I'll use this as some time to consider other very different alternatives that do not require an architecture change:

• Pull, without a generalized solution: Integrations can provide the above rest API themselves, exposed for being consumed by their Add-on service. The integration would be responsible for understanding the current state of camera streams, and can call the existing camera method to return a stream source for a Camera entity to implement this: async def async_get_stream_source(hass: HomeAssistant, entity_id: str) -> str | None. This method is currently only used by homekit for providing access to rstp source, as well as unit tests.
• Push, without a generalized solution: Integrations push to the external services. The external services implement their own APIs for adding/updating streams similar to what RTSPtoWeb has for example. There is additional complexity here such as how authentication/security would work, etc that would need to be considered for each integration.
• Explore even more generalized protocols/APIs: Not sure what prior art exists here. My impression is Matter will not necessarily be targeting cameras in the first round (so, many years away). I'm not actively looking at this at the moment.

With respect to security, I like the pull model since it uses Home Assistant authentication to access the streams. We'd also assume existing security controls for the stream are OK (e.g. credentials in the url, or expiring tokens) when handing over to the service, but if there are options for finer grained access controls this might be worth consideration here. (e.g. my prometheus service scraping entity state doesn't need access to camera streams)

[balloob]

So Home Assistant currently does not have APIs that expose integration credentials.

If this feature request is mainly driven by Frigate, wouldn't it make more sense that Frigate just reads the HA config files and parses the required information ?

[allenporter]

The interface between camera and stream is currently basically just a url. That url may or may not contain credentials. I'm not sure by your comment if you are referring to a url that contains credentials or something else e.g. auth headers. (If you're referring to alternative #2 i referenced, what i meant was the add-on authenticating home assistant and allowing it to update stream sources. I wrote that before understanding the add-on security model.)

My impression is that HA config files don't always contain the necessary information for rtsp streams e.g. cloud based cameras without stable urls like nest. HA config files do contain necessary information to totally re-create the integration from an add-on, however, in theory.

My current leaning is towards a not extending the home assistant architecture at all, and having a custom component exposing RTSP stream urls to its own add-on (alternative #1). So i consider this a dead proposal now.

Generally, I wonder what the future of the open home looks like when aggregators of devices know how to share devices with other. I'm not at all an expert on that topic and get the impression its not an immediate problem to solve today, and i'm guessing the solution probably would not involve narrow rest apis.

[hunterjm]

I don't know if I would necessarily consider this integration credentials, though I guess it could be for standard RSTP feeds which require credentials to access. The main use case here would be for Nest or other integrations that have a rotating RTSP feed that expires and gets refreshed every 5 minutes or so. This provides a way for Home Assistant to be the source of the camera integration and allow third parties to ingest Home Assistant cameras directly.

Another way this can be achieved is by using the HLS stream provided by the Home Assistant API today, but that introduces additional delay and uses custom tokens for authentication and determining what stream to return. I know you replaced some of the custom tokenization in some integrations though, and we could do the same here by re-working the HLS API to use the entity name and sign the URLs.

Edit: This would be more in line with what other parts of the API do today, but as mentioned would make Home Assistant the "middle man" as all traffic for streams would flow through it instead of to the cameras directly.

[BeardyBrah]

Seconding this / bump.

Use case: Installing the Nest integration provided me with a camera entity as 'camera.front_door' however this can't be used in Frigate for local NVR and object detection.

[felipecrs]

I just came to know this proposal now. In fact, I had even proposed and implemented the very same thing at home-assistant/core#81755, which got rejected.

Then I created a custom integration for it: https://github.com/felipecrs/hass-expose-camera-stream-source

And it has been helping several users since then.

[parautenbach]

I just came to know this proposal now. In fact, I had even proposed and implemented the very same thing at home-assistant/core#81755, which got rejected.

Commenting on:

Never mind. Services apparently can't return data when called.

Services can now return data via the response variable.

[felipecrs]

Oh yes. Since this ability was released I have been thinking about converting the integration to add a new service instead:

• Create a service that responds with the camera stream URL felipecrs/hass-expose-camera-stream-source#34.

If having this as a service call helps changing @balloob's opinion about my PR, I am happy to propose a PR again.

No pressure though from my side. The custom integration works great.