Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Live Streams Component #21473

Merged
merged 20 commits into from Mar 12, 2019

Conversation

Projects
None yet
10 participants
@hunterjm
Copy link
Contributor

hunterjm commented Feb 27, 2019

Description:

This MVP provides a stream component which enables cameras with a new stream_source property to generate a HLS stream that is available for 5 minutes from the time it is requested and/or last accessed. Initially, there is no transcode support for video formats, so the input source must be H264 Annex-B format (which my HikVision camera RTSP streams are).

The component was written in a way so that video segments are kept in memory, and CPU consumption is very low. Other formats can be added with relative ease in the future, as well as enabling transcoding.

The frontend can access these streams via the WebSocket request, and display them across all major browsers with the help of hls.js. There is also a new service introduced that will call play_media with the stream on a media_player device that supports HLS. The immediate use case here is being able to stream your live camera to a Google Cast device.

Related issue (if applicable): home-assistant/architecture#113

Pull request in home-assistant.io with documentation (if applicable): home-assistant/home-assistant.io#8838

Example entry for configuration.yaml (if applicable):

stream:

Checklist:

  • The code change is tested and works locally.
  • Local tests pass with tox. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • New dependencies have been added to the REQUIREMENTS variable (example).
  • New dependencies are only imported inside functions that use them (example).
  • New or updated dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
  • New files were added to .coveragerc.

If the code does not interact with devices:

  • Tests have been added to verify that the new code works.
Show resolved Hide resolved tests/components/stream/common.py Outdated
Show resolved Hide resolved tests/components/stream/common.py Outdated
Show resolved Hide resolved tests/components/stream/common.py Outdated
Show resolved Hide resolved tests/components/stream/common.py Outdated
Show resolved Hide resolved tests/components/stream/common.py
@hunterjm

This comment has been minimized.

Copy link
Contributor Author

hunterjm commented Feb 27, 2019

Travis build is going to fail because pyAV needs FFmpeg installed to get the libav bindings. I'll work on updating the travis config.

@Kane610

This comment has been minimized.

Copy link
Contributor

Kane610 commented Feb 27, 2019

I think that it's fine to drop support for mjpeg completely. I don't see us wanting to use this once we can do hls.

@balloob there are a lot of cameras not supporting h264 still being used with home assistant, removing jpeg would be bad for a lot of users

@hunterjm hunterjm requested a review from home-assistant/docker as a code owner Feb 27, 2019

@hunterjm

This comment has been minimized.

Copy link
Contributor Author

hunterjm commented Feb 27, 2019

I think that it's fine to drop support for mjpeg completely. I don't see us wanting to use this once we can do hls.

@balloob there are a lot of cameras not supporting h264 still being used with home assistant, removing jpeg would be bad for a lot of users

@Kane610 - He was making a commit comment on an mjpeg.py stream provider I had created. It was duplicating the work being done by the camera_proxy_stream endpoint, but using the new stream component. I don't think we are planning on dropping MJPEG support, rather the frontend will determine if an HLS stream is available via the websocket call, with the camera_proxy_stream MJPEG as the fallback.

@robbiet480

This comment has been minimized.

Copy link
Member

robbiet480 commented Feb 28, 2019

I think the component being named stream is a bit too generic, I'd suggest video_stream.

@balloob

This comment has been minimized.

Copy link
Member

balloob commented Feb 28, 2019

Went through the code, it's beautiful. Haven't had time to test it today, but now I have something to look forward to tomorrow 👍

@balloob

This comment has been minimized.

Copy link
Member

balloob commented Feb 28, 2019

Alright, have been doing some local testing. I don't know if it's my Mac/ffmpeg or camera (using UVC G3 Flex), but there is quite some delay.

Also at the end of the stream my Google Home got some bad login attempts. I don't think that we should mark them as bad login attempts.

2019-02-28 10:48:43 INFO (MainThread) [homeassistant.components.http.view] Serving /api/hls/affe09662045eae927c309e2d6865930386841e6b93e4075ff941e111809dd16/playlist.m3u8 to 192.168.1.58 (auth: False)
2019-02-28 10:48:46 WARNING (stream_worker) [libav.aac] 2 frames left in the queue on closing
2019-02-28 10:48:46 WARNING (stream_worker) [libav.mpegts] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
2019-02-28 10:48:46 WARNING (stream_worker) [libav.mpegts] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
2019-02-28 10:48:49 WARNING (stream_worker) [libav.aac] 2 frames left in the queue on closing
2019-02-28 10:48:49 INFO (MainThread) [homeassistant.components.http.view] Serving /api/hls/affe09662045eae927c309e2d6865930386841e6b93e4075ff941e111809dd16/playlist.m3u8 to 192.168.1.58 (auth: False)
2019-02-28 10:48:49 WARNING (MainThread) [homeassistant.components.http.ban] Login attempt or request with invalid authentication from 192.168.1.58
2019-02-28 10:48:50 INFO (MainThread) [homeassistant.components.http.view] Serving /api/hls/affe09662045eae927c309e2d6865930386841e6b93e4075ff941e111809dd16/playlist.m3u8 to 192.168.1.58 (auth: False)
2019-02-28 10:48:50 WARNING (MainThread) [homeassistant.components.http.ban] Login attempt or request with invalid authentication from 192.168.1.58
2019-02-28 10:48:51 INFO (MainThread) [homeassistant.components.http.view] Serving /api/hls/affe09662045eae927c309e2d6865930386841e6b93e4075ff941e111809dd16/playlist.m3u8 to 192.168.1.58 (auth: False)
2019-02-28 10:48:51 WARNING (MainThread) [homeassistant.components.http.ban] Login attempt or request with invalid authentication from 192.168.1.58
2019-02-28 10:48:52 INFO (MainThread) [homeassistant.components.http.view] Serving /api/hls/affe09662045eae927c309e2d6865930386841e6b93e4075ff941e111809dd16/playlist.m3u8 to 192.168.1.58 (auth: False)
2019-02-28 10:48:52 WARNING (MainThread) [homeassistant.components.http.ban] Login attempt or request with invalid authentication from 192.168.1.58
@balloob

This comment has been minimized.

Copy link
Member

balloob commented Feb 28, 2019

I takes ~30 seconds for me to get my stream on my Google Home. Video has ~25 seconds delay. I don't know if that's because of my camera or the component.

(I was using the medium stream, switched to the high stream and it dropped to 20s)

@balloob

This comment has been minimized.

Copy link
Member

balloob commented Feb 28, 2019

We should listen to the homeassistant_stop event. I get this exception when I close Home Assistant while streaming:

Exception in thread stream_worker:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/paulus/dev/hass/home-assistant/homeassistant/components/stream/worker.py", line 91, in stream_worker
    )), hass.loop)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py", line 809, in run_coroutine_threadsafe
    loop.call_soon_threadsafe(callback)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 717, in call_soon_threadsafe
    self._check_closed()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 461, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py:951: RuntimeWarning: coroutine 'StreamOutput.put' was never awaited
  del exc_type, exc_value, exc_tb
@balloob

This comment has been minimized.

Copy link
Member

balloob commented Feb 28, 2019

Also, we might want to silence libav warnings in prod.

@balloob

This comment has been minimized.

Copy link
Member

balloob commented Feb 28, 2019

Opening a second time is near instant to open the stream (as the stream was still open), it still has a delay .

@balloob

This comment has been minimized.

Copy link
Member

balloob commented Feb 28, 2019

Can we print a log line when we start/stop a stream worker?

@@ -1,8 +1,18 @@
sudo: false
dist: xenial

This comment has been minimized.

Copy link
@awarecan

awarecan Feb 28, 2019

Contributor

Will this change all of our py 3.5 and 3.6 build?

This comment has been minimized.

Copy link
@hunterjm

hunterjm Mar 1, 2019

Author Contributor

Yes, it will change all builds to run on Ubuntu 16.04.

addons:
apt:
sources:
- sourceline: "ppa:jonathonf/ffmpeg-4"

This comment has been minimized.

Copy link
@awarecan

awarecan Feb 28, 2019

Contributor

How much time will this add to build proces? I fell uncomfortable to add more loads to our already slow build pipeline. Will it better if we create a video test build config to only run the test which depends on ffmpeg and/or other video related libs.

This comment has been minimized.

Copy link
@hunterjm

hunterjm Mar 1, 2019

Author Contributor

It does add a minute or so to the build to install these packages. The other options are to skip stream tests, or move CI platforms.

Show resolved Hide resolved homeassistant/components/camera/__init__.py
Show resolved Hide resolved homeassistant/components/camera/__init__.py Outdated
@hunterjm

This comment has been minimized.

Copy link
Contributor Author

hunterjm commented Mar 1, 2019

I takes ~30 seconds for me to get my stream on my Google Home. Video has ~25 seconds delay. I don't know if that's because of my camera or the component.

(I was using the medium stream, switched to the high stream and it dropped to 20s)

The delay is for a couple of reasons, the initial delay is opening the stream and starting the worker thread, then waiting for the first segment to be ready before serving the endpoint. The video delay is due to HLS requiring a buffer of video segments before starting playback. Without that buffer, you would get a "buffering" wheel consistently after every segment.

The delay is dependent on the length of the segments. If you manually open the .m3u8 playlist, you will see a #EXT-X-TARGETDURATION line. The value there is the keyframe (or "I" frame) interval. We mux a new segment on every video keyframe. The way most encoders work is one frame will have the entire image, and subsequent frames will only contain diffs (to keep size down). Unless we transcode the video, we can not re-define the keyframe interval.

HikVision cameras let you set this in the camera options. My cameras are set to 12 FPS with a I Frame interval of 30 frames, which gives me a keyframe every 2.5 seconds, so I get 2.5 second segments. The default HLS provider I have in this PR returns the 3 most recent segments. For me, this means a maximum delay of 8 seconds + load and network latency, so real world delay of 8-10 seconds. If you are seeing a 20s delay, I would imagine your per-segment duration is ~6 seconds. I do not know if that can be configured in UniFi cameras.

What can be changed is this component can make the number of segments to persist and serve configurable, with a default of 3. The user can work backwards from there until their network conditions give them buffering issues to reduce delay.

Opening a second time is near instant to open the stream (as the stream was still open), it still has a delay.

Near instant opening because the worker was already started. I want to add a "preload" config option that would open and begin processing streams on Home Assistant startup eventually. The question is where to put it. A per-camera config option or global stream config?

Also, we might want to silence libav warnings in prod.

I agree here. I silenced them manually in my logger component. Not sure how to do that in code, or where we would want to put it.

@awarecan
Copy link
Contributor

awarecan left a comment

Not look test yet, I will call it a day.

Show resolved Hide resolved homeassistant/components/camera/services.yaml
Show resolved Hide resolved homeassistant/components/stream/__init__.py
Show resolved Hide resolved homeassistant/components/stream/__init__.py Outdated
Show resolved Hide resolved homeassistant/components/stream/__init__.py Outdated
Show resolved Hide resolved homeassistant/components/stream/__init__.py Outdated
Show resolved Hide resolved homeassistant/components/stream/core.py
Show resolved Hide resolved homeassistant/components/stream/hls.py Outdated
Show resolved Hide resolved homeassistant/components/stream/hls.py Outdated
Show resolved Hide resolved homeassistant/components/stream/hls.py Outdated
Show resolved Hide resolved homeassistant/components/stream/hls.py Outdated

hunterjm added some commits Feb 27, 2019

@hunterjm hunterjm force-pushed the hunterjm:feature/streams branch from 029f7a1 to c66b834 Mar 8, 2019

@hunterjm

This comment has been minimized.

Copy link
Contributor Author

hunterjm commented Mar 8, 2019

@balloob - I rebased, but still am not seeing GH actions checks. Will probably need to update the actions containers to include the libs for pyAV to install correctly though. Should be the same modifications made in virtualization/Docker/setup_docker_prereqs.

@hunterjm hunterjm changed the title WIP: Live Streams Component Live Streams Component Mar 8, 2019

Show resolved Hide resolved homeassistant/components/camera/__init__.py Outdated
Async friendly.
"""
from homeassistant.components.stream import request_stream

This comment has been minimized.

Copy link
@balloob

balloob Mar 9, 2019

Member

This still need to be addressed

Show resolved Hide resolved homeassistant/components/camera/__init__.py Outdated
Show resolved Hide resolved homeassistant/components/camera/__init__.py Outdated
Show resolved Hide resolved homeassistant/components/camera/__init__.py Outdated
from homeassistant.components.media_player.const import (
ATTR_MEDIA_CONTENT_ID, ATTR_MEDIA_CONTENT_TYPE,
SERVICE_PLAY_MEDIA, DOMAIN as DOMAIN_MP)
from homeassistant.components.stream import request_stream

This comment has been minimized.

Copy link
@balloob

balloob Mar 9, 2019

Member

It's fine to import files and functions. You already import OUTPUT_FORMATS etc.

Update your patch to be patch('homeassistant.components.camera.request_stream')

Show resolved Hide resolved homeassistant/components/camera/__init__.py Outdated
Show resolved Hide resolved homeassistant/components/stream/__init__.py
Show resolved Hide resolved homeassistant/components/stream/core.py Outdated

hunterjm added some commits Mar 11, 2019

implement registry and have output provider remove itself from stream…
… after idle, set libav log level to error

@balloob balloob merged commit 7ccd0bb into home-assistant:dev Mar 12, 2019

4 checks passed

Hound No violations found. Woof!
cla-bot Everyone involved has signed the CLA
continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.07%) to 92.825%
Details

@wafflebot wafflebot bot removed the in progress label Mar 12, 2019

@balloob

This comment has been minimized.

Copy link
Member

balloob commented Mar 12, 2019

CC @pvizeli let's make sure this builds on Hass.io 👍

@hunterjm hunterjm deleted the hunterjm:feature/streams branch Mar 13, 2019

@danjenkins

This comment has been minimized.

Copy link

danjenkins commented Mar 19, 2019

Not a comment on the code but curious if theres still a delay in the stream and what we can do to combat that? I used HLS with nginx from a RTSP stream and found the delay made it useless - would love this feature but not if we're just going to see the same result by different means - I too am using unifi video like @balloob.

@MartinHjelmare

This comment has been minimized.

Copy link
Member

MartinHjelmare commented Mar 19, 2019

Please open an issue if you suspect a bug.

If you want to suggest an enhancement please open a feature request in the Feature Requests section of our community forum.

Merged PRs should not be used for enhancement discussion or bug reports. If you've found a bug it's ok to make a review with inline comments and link to an issue that reports the bug.

Thanks!

@home-assistant home-assistant locked as resolved and limited conversation to collaborators Mar 19, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.