Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Miscellaneous Arlo warnings and errors #13176

Closed
arsaboo opened this issue Mar 13, 2018 · 33 comments · Fixed by #14823
Closed

Miscellaneous Arlo warnings and errors #13176

arsaboo opened this issue Mar 13, 2018 · 33 comments · Fixed by #14823

Comments

@arsaboo
Copy link
Contributor

arsaboo commented Mar 13, 2018

Home Assistant release (hass --version): 0.65.3

Python release (python3 --version): 3.6.2

Component/platform: Arlo

Description of problem: Looks like our Arlo platform needs some TLC. Currently, the the Arlo integration slows down the system and generates lots of errors/warnings as below:

2017-11-15 08:35:03 WARNING (MainThread) [homeassistant.setup] Setup of alarm_control_panel is taking over 10 seconds.
2017-11-15 08:35:03 WARNING (MainThread) [homeassistant.setup] Setup of camera is taking over 10 seconds.
2017-11-15 08:35:05 WARNING (MainThread) [homeassistant.components.alarm_control_panel] Setup of platform arlo is taking over 10 seconds.
2017-11-15 08:35:05 WARNING (MainThread) [homeassistant.components.camera] Setup of platform arlo is taking over 10 seconds.

image

These are not new issues and have been reported several times on the forums
https://community.home-assistant.io/t/insteon-hub-pro-help/31276/8
https://community.home-assistant.io/t/hassio-io-10-second-warnings/23714/11

I did some digging and added So, I added _LOGGER.debug("Arlo component update") after this statement and _LOGGER.debug("Arlo sensor update") after this one and here is the log:

2018-03-12 20:34:36 DEBUG (Thread-7) [custom_components.arlo] Arlo component update
2018-03-12 20:34:37 DEBUG (Thread-2) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-9) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-3) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-10) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-11) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-4) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-6) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-12) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-13) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-5) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-14) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-8) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-19) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-20) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-21) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-13) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-18) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-4) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-8) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-2) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-7) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-9) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-12) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-14) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-11) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-3) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-15) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-16) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-17) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-5) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-13) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-18) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-6) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-7) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-12) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-21) [custom_components.sensor.arlo] Arlo sensor update

Looks like each sensor has its own log entry, suggesting that every sensor has its own Arlo object that it updates every time. We need to change that so that all sensors use one Arlo data object, which is updated every 30 seconds or so.

cc: @tchellomello

@tchellomello
Copy link
Contributor

Adding myself to monitor this issue

@jwillaz
Copy link
Contributor

jwillaz commented Mar 13, 2018

I think I figured out a solution. I tested these changes directly. Don't have time at the moment to set up Git and do PR right now, but if someone wants to run with these changes.

from homeassistant.util import Throttle

Fix for battery drain:
SCAN_INTERVAL = timedelta(seconds=3600)

Add above "def update(self)" in camera/arlo.py and sensor/arlo.py:
@Throttle(SCAN_INTERVAL) 

ALSO, @tchellomello, the logic I added in base_station.py in the Pyarlo library for doing makeshift throttling should be removed:

current_time = int(time.time())
last_refresh = 0 if self._last_refresh is None else self._last_refresh

if current_time >= (last_refresh + self._refresh_rate):

Since we'll be using @Throttle, _refresh_rate and _last_refresh should not be needed. Apologies for my poor understanding of how throttling works in home assistant. If there is a better way to handle this, by all means go for it.

My only concern is making sure each call to an individual camera via either the Camera component or Sensor component isn't cascading via the Base Station into multiple calls to ALL cameras. I think @Throttle on update should help with that. SCAN_INTERVAL being set to 1 hour (especially for the Sensor component) will help with battery drain because having it set to 90 seconds means that the base station is waking up each camera every 90 seconds for info updates. Camera settings and values (like signal strength, battery status, etc) shouldn't be updating that often.

@jwillaz
Copy link
Contributor

jwillaz commented Mar 13, 2018

There may be something additional at play here. After a while, I started receiving the "____ is taking over 10 seconds" warnings from both sensor and camera components.

@arsaboo
Copy link
Contributor Author

arsaboo commented Mar 17, 2018

It seems Arlo has completely stopped working off late. Also, see https://community.home-assistant.io/t/arlo-cameras-and-sensors-entities-not-showing-up/33684/16

@JBenzie
Copy link

JBenzie commented Mar 19, 2018

I've also noticed an issue with my Arlo camera. I've been receiving the various warnings in the logs as stated above, however, today it stopped working altogether. There are now log entries stating each component is failing.

@mwd102
Copy link

mwd102 commented Mar 25, 2018

New to Hass and an Arlo owner, having the same issues. I think I have a fix for the connection pool discards, PR here: #13445

Short of it is pyarlo is not designed with asynchronicity in mind - increased thread pool and set threads to block so that requests coming in when a pool is full (happens often, thanks to netgears slow response times) it waits instead of discarding.

@mwd102
Copy link

mwd102 commented Mar 25, 2018

Taking a look into the updating -

Each sensor has a reference to the same PyArlo and thus the same ArloBaseStation (so far so good).

The issue is with the async code, the update method will be called from however many sensors the user has. The check for the last_refresh will eval to true partly because the time is rounded to an int and partly because of the time it takes to make the web request and set the last update before other threads get there.

A hacky fix to prove the concept was to set a lock in the update function in base_station.py of the pyarlo module:

def update(self):
        if self._update_lock:
            while self._update_lock:
                time.sleep(0.5)
                if not self._update_lock:
                    return self.camera_properties

        self._update_lock = True
        current_time = time.time()

        last_refresh = 0 if self._last_refresh is None else self._last_refresh

        if current_time >= (last_refresh + self._refresh_rate):
            self.get_cameras_properties()
            self._attrs = self._session.refresh_attributes(self.name)
            _LOGGER.debug("Called base station update of camera properties: "
                          "Scan Interval: %s, New Properties: %s",
                          self._refresh_rate, self.camera_properties)
        self._update_lock = False

this means only the first sensor to ask for the update will succeed in a web request. in order to solve the issue of returning up to date info to all the others, i just parked them with a sleep and return when the thread that is making the request updates the data and releases the lock. like i said, PoC - interested in others thoughts?

@jwillaz
Copy link
Contributor

jwillaz commented Mar 27, 2018

@mwd102 - you are correct about there being issues in the update methods. I originally added that timing logic as a rudimentary way of reducing the number of calls made to the Arlo endpoints. It worked in that regard, and obviously sometimes you would still get multiple calls because there was no real locking mechanism. I have, however, been using the following modified code to tighten that up a bit more:

def update(self):
        """Update object properties."""
        current_time = int(time.time())
        last_refresh = 0 if self._last_refresh is None else self._last_refresh

        with self._update_lock:
            if current_time >= (last_refresh + self._refresh_rate):
                self.get_cameras_properties()
                _LOGGER.debug("Called base station update of camera properties: "
                              "Scan Interval: %s, New Properties: %s",
                              self._refresh_rate, self.camera_properties)
                self._session.refresh_attributes()
                self._last_refresh = int(time.time())
                for device in self._session._device_data:
                    if device.get('deviceName') == self.name:
                        self._attrs = device
                        return
        return

Even with these modifications, I still receive the warnings/errors in question. Your approach may be a better one. I am very novice with Python, but I love HASS and really appreciate the work folks like @tchellomello have done to get the Arlo component started, especially with no official API support from Netgear. I hope more people, especially those with stronger Python skills than mine, will get involved.

From my perspective, the biggest thing causing issues right now is the difference in concept of a "camera" entity in HASS vs what that means in Arlo systems. Typically, a Camera would be some singularly addressable entity with its own endpoint, attributes, stream, JPEG captures, etc. This is sort of the case with standalone Arlo cameras like Arlo Q, Arlo Baby, and Arlo Go, but not so much with Arlo and Arlo Pro which are dependent on the base station. Each of the "camera" entities for Arlo and Arlo Pro is really just a mock-up of data being retrieved from the Base Station. A single call for update to the Base Station will contain most of the new information for ALL cameras, which is why I wanted to throttle these update calls in the first place: pending you have more than one camera, more than one call to the base station is unnecessary. And then there are "other" camera attribute update calls on top of that. Confusing.

@mwd102
Copy link

mwd102 commented Mar 27, 2018

@jwillaz You're totally right - a the concept of camera in pyarlo (for Pro at least) and sensor in HASS are not equivalent which is causing issues. I need to look into the HASS state machine and signalling to see if we can come up with something better. Have the base station call update() and cascade to the sensor 'camera'.

The other errors you are getting might be fixed by my other suggestion in the PR which is below. HASS rejected that as they feel (rightly) that this is probably better placed upstream. Try adding the below onto line 46 of components/arlo.py

(from requests.adapters import HTTPAdapter)

arlo.session.mount('https://',HTTPAdapter(pool_maxsize=10,pool_block=True)) 

I don't see any indication that I am making more than one request with those two patches, neither do I get too slow warns or connection discards. Totally accept this is probably not a finished solution. Would be really interested in @tchellomello thoughts on wether we should update pyarlo or work something into HASS.

@jurgenweber
Copy link

jurgenweber commented Apr 10, 2018

I have an Arlo Baby and nothing works.. Someitmes when I restart the camera might show one photo but all of the sensors are blank. Would this be related or another problem and I should make another issue?

the interesting part is I have no logs... even after adding this to my logger config:

logger:
  default: warn
  logs:
    homeassistant.components.automation: info
    homeassistant.components.arlo: info
    homeassistant.components.camera.arlo: info
    homeassistant.components.sensor.arlo: info

@codypet
Copy link

codypet commented Apr 17, 2018

@jurgenweber I'm having the exact same problem. I believe its related because I'm getting the warnings under info. I was starting to have these problems when I went to 0.65.6, but the kicker is that it started working fine randomly until I updated to release 0.67. I hadn't done anything to my Hass in 2 weeks and then one day it just started working. I'm wondering if the same thing will happen again in a week.

@jurgenweber
Copy link

yeah, I still get nothing. Shame no one else in the thread has piped up. I will debug more in the near future.

@jurgenweber
Copy link

ok, after some investigation...... I found out:

tchellomello/python-arlo#73

but I feel that this project is now a littler dead..... I then found this one; https://github.com/jeffreydwalter/arlo which seems to be much more active.

I am thinking we should swap to the more active project for the integrations and move on, etc. If I have some time I will look into that deeper.

@jeffreydwalter
Copy link

@jurgenweber I do actively maintain my library and welcome any contributions.

I am happy to do what I can to add support for any missing features in my library. I only own the first generation Arlo cameras, so any features not directly related to those cameras will require assistance from others to add and test. That being said, my library does work for Arlo, Arlo 2, Arlo Pro, Arlo Q, and Arlo GO. I'm sure there are probably some minor features that are not currently exposed via the API in my library, but when they are brought to my attention I do my best to add them.

@arsaboo
Copy link
Contributor Author

arsaboo commented May 1, 2018

@jurgenweber @jeffreydwalter Can this new library be used in a custom component right now?

@tchellomello
Copy link
Contributor

Hey guys, it's been a while since I last touch on the pyarlo lib. @jeffreydwalter how about if we merge both projects and rebase the HA component using only one project?

@rafuz
Copy link

rafuz commented May 20, 2018

Hello guys, any news on this topic?

@jurgenweber
Copy link

I started and I have on my machine some sensors using @jeffreydwalter library but my boy was born and I have no time. The sensors were taking too long to update giving a warning and I wanted to look into any alternative approach for updating them instead of querying arlo.netgear.com every time but by actually listening to the stream from the arlo and updating the sensors on the fly..... I am finding it hard to work out the HaSS eco system as this is the first time I have tired to do something in it as well. .... :\ I would really love something thou.

@tchellomello
Copy link
Contributor

Guys, I'm going to work on this issue this week and see if we can do anything to address that.

@tchellomello
Copy link
Contributor

Guys, I've started working today on a new PR (still local only) that fixed this problem. Actually, we are using now some cache states which will reduce a lot of queries to the Arlo API.

Just give me a few more days and I'll submit a PR with some test instructions so then can you test it too.

@jurgenweber
Copy link

jurgenweber commented May 29, 2018

yay, looking forward to it and I am happy to test.

@grantclem
Copy link

also happy to test, thanks for your effort resolving this

@jwillaz
Copy link
Contributor

jwillaz commented May 31, 2018

@tchellomello Do you know how to make the SCAN_INTERVAL dynamic (set in yaml config) for the camera & sensor components? Right now they are set to 90 seconds, but for battery-operated Arlos, that's too frequent and causes noticeable battery drain (I've seen 2-5% depletion per day). I've tested with the interval set to 1 hour, and it has been fine. Granted, that would not be optimal for reading property changes made outside of HASS (like updating video quality), but ideally functionality for controlling those settings would eventually get baked into this component so we don't have to rely so heavily on polling data.

@tchellomello
Copy link
Contributor

@jwillaz I'm working on a PR that will change this behavior and get the data polling much better. I'm almost done with the new refactoring and I'll let you know as soon I get it published for testing.

@MuppetOwl
Copy link

Great news the Arlo component get some love :) I have a Arlo Q plus and would be great to be able to activate the motion detection from Home Assistant and also see the status for the sensors like wifi signal etc (They dont work on the Q now, just unknown status). Any plans to support more function for the Arlo Q? I can help if you need someone to test.

@tchellomello
Copy link
Contributor

tchellomello commented Jun 2, 2018

Guys, started some work at tchellomello/python-arlo#75 and potential changes to the HASS components here tchellomello/home-assistant@dev...arlo_fix

It's been working for me now but I want to leave it running for 1 or 2 days to see how it goes.

I'll keep you posted.

@jwillaz
Copy link
Contributor

jwillaz commented Jun 3, 2018

@tchellomello Mostly good, so far! I'm not seeing the timeouts that typically occur after several hours and the constant sensor warnings for taking too long. The one warning I receive on HASS startup is this:

Updating state for camera.xxxxxx (<class 'homeassistant.components.camera.arlo.ArloCam'>) took 1.890 seconds. Please report platform to the developers at https://goo.gl/Nvioub

Other than that, I am still seeing some steep battery drain on the cameras (2% in 12 hours). I checked queries from the Arlo hub in Pi-Hole, and it's sending out two queries every 2 minutes.

@grantclem
Copy link

Hi, just wondering how I go about testing this tchellomello/home-assistant@dev...arlo_fix? Custom_components?

@grantclem
Copy link

grantclem commented Jun 5, 2018

Hi, using Alok's repo I got it working, Few errors and warnings still coming up. But essentially working.

2018-06-04 23:22:33 WARNING (MainThread) [homeassistant.helpers.entity] Update of alarm_control_panel.base_station is taking over 10 seconds

2018-06-04 23:26:00 WARNING (MainThread) [homeassistant.components.alarm_control_panel] Updating arlo alarm_control_panel took longer than the scheduled update interval 0:00:30

2018-06-04 23:26:42 ERROR (MainThread) [homeassistant.core] Error doing job: Future exception was never retrieved
Traceback (most recent call last):
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/homeassistant/.homeassistant/custom_components/arlo.py", line 83, in hub_refresh
    update_base_station=True)
  File "/srv/homeassistant/lib/python3.6/site-packages/pyarlo/__init__.py", line 262, in update
    base.update()
  File "/srv/homeassistant/lib/python3.6/site-packages/pyarlo/base_station.py", line 457, in update
    self._attrs = self._session.refresh_attributes(self.name)
  File "/srv/homeassistant/lib/python3.6/site-packages/pyarlo/__init__.py", line 206, in refresh_attributes
    data = self.query(url).get('data')
AttributeError: 'NoneType' object has no attribute 'get'

@tchellomello
Copy link
Contributor

tchellomello commented Jun 5, 2018

@grantclem thanks for the report. This issue has been fixed and will be included in the next version.

2018-06-04 23:26:42 ERROR (MainThread) [homeassistant.core] Error doing job: Future exception was never retrieved
Traceback (most recent call last):
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/homeassistant/.homeassistant/custom_components/arlo.py", line 83, in hub_refresh
update_base_station=True)
File "/srv/homeassistant/lib/python3.6/site-packages/pyarlo/init.py", line 262, in update
base.update()
File "/srv/homeassistant/lib/python3.6/site-packages/pyarlo/base_station.py", line 457, in update
self._attrs = self._session.refresh_attributes(self.name)
File "/srv/homeassistant/lib/python3.6/site-packages/pyarlo/init.py", line 206, in refresh_attributes
data = self.query(url).get('data')
AttributeError: 'NoneType' object has no attribute 'get'

See: tchellomello/python-arlo#76

@tchellomello
Copy link
Contributor

Guys, PR #14823 created. It's still flagged as WIP since I have to update the documentation but any feedback is very welcome.

Thanks

@jurgenweber
Copy link

woohoo! Does that mean it will be in 0.71.1 or 0.72.0?

Thanks

@tchellomello
Copy link
Contributor

0.72.0

@home-assistant home-assistant locked and limited conversation to collaborators Oct 26, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet