Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add WebRTC stream player #10193

Merged
merged 11 commits into from
Oct 13, 2021
Merged

Add WebRTC stream player #10193

merged 11 commits into from
Oct 13, 2021

Conversation

allenporter
Copy link
Contributor

Breaking change

Proposed change

Add support for playing WebRTC Streams, initially motivated by Nest Battery Camera / Doorbell.
This is an implementation of home-assistant/architecture#640

This introduces a new element ha-web-rtc-player that is used by the ha-camera-stream element for cameras of stream_type for STREAM_TYPE_WEB_RTC.

The stream is initiated by creating a RTCPeerConnection which creates an offer. This is passed down to core, where the integration is asked to pass the offer to the remote device via a signaling path, and the response is an answer which contains information for how the javascript client can connect to the remote device. See https://webrtc.org/getting-started/peer-connections for additional detail on establishing a connection. In the specific case of a nest camera, this translates to a GenerateWebRtcStream RPC against the API.

Once a connection is established, the remote tracks are added to the local MediaStream for playback. See https://webrtc.org/getting-started/remote-streams for more detail. Nest cameras require a data channel, however, it does not appear to actually communicate any information at the moment.

Overall tasks I have planned:

  • Initial working end-to-end, tested on Chrome Desktop with a Nest Doorbell Camera
  • Handle resizing of the window for vertical video for Doorbell cameras
  • Testing on multiple browsers and devices
  • Testing of various NAT scenarios
  • Error handling for initial connection failures
  • Handling for errors once stream is established (e.g. nest stream expiration)

Support for nest WebRTC is tracked in home-assistant/core#55302

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New feature (thank you!)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Example configuration

Additional information

Checklist

  • The code change is tested and works locally.
  • There is no commented out code in this PR.
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

Comment on lines 136 to 139

video {
width: 100%;
}
Copy link
Member

@balloob balloob Oct 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

block defaults to width: 100%;

Suggested change
video {
width: 100%;
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This actually is not a no-op -- however, i am ok with starting over on the CSS given it doesn't render vertical video right yet. When taking this away, the video get wider than the dialog. (I think the default video quality is pretty massive.) hls and the camera itself also have a similar block for what its worth.

src/data/camera.ts Outdated Show resolved Hide resolved
src/data/camera.ts Outdated Show resolved Hide resolved
@allenporter
Copy link
Contributor Author

Also, i'll take a pass at applying these suggestions to the existing hls camera where this was copied from. Thank you.

supportsFeature(this.stateObj, CAMERA_SUPPORT_STREAM) &&
this.stateObj.attributes.stream_type === STREAM_TYPE_WEB_RTC
) {
if (typeof RTCPeerConnection === "undefined") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we fallback to mjpeg in this case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The only flaw I find here is that this basically makes it a silent failure. In the case of nest webrtc for example, mjpeg isn't supported so it just sends an image which doesn't end up loading leaving an empty dialog box.

Perhaps in a follow up PR I can add an onerror for the image load. Then display the series of errors (web rtc not supported, failed to fetch stream url, failed to load mjpeg url). Is that a worthwhile improvement/idea?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think so.

@bramkragten bramkragten merged commit 4ad005f into home-assistant:dev Oct 13, 2021
@github-actions github-actions bot locked and limited conversation to collaborators Oct 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants