Update sensor to only query Glow API shortly after 0 and 30 mins past the hour #136

jackyaz · 2022-02-08T18:36:30Z

This is to address Glow's concerns referenced in #126

There's probably a better way to do this, but it seems to work. I'm new to HA development so finding my feet a little.

This does cause HA to complain that the sensor update is taking over 10s, but doesn't seem to be a problem

Feb 08 18:30:00 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:00 DEBUG (MainThread) [custom_components.hildebrandglow_dcc.sensor] Update time, sleeping 22 before talking to API
Feb 08 18:30:00 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:00 DEBUG (MainThread) [custom_components.hildebrandglow_dcc.sensor] Update time, sleeping 30 before talking to API
Feb 08 18:30:00 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:00 DEBUG (MainThread) [custom_components.hildebrandglow_dcc.sensor] Update time, sleeping 77 before talking to API
Feb 08 18:30:00 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:00 DEBUG (MainThread) [custom_components.hildebrandglow_dcc.sensor] Update time, sleeping 2 before talking to API
Feb 08 18:30:00 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:00 DEBUG (MainThread) [custom_components.hildebrandglow_dcc.sensor] Update time, sleeping 7 before talking to API
Feb 08 18:30:00 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:00 DEBUG (MainThread) [custom_components.hildebrandglow_dcc.sensor] Update time, sleeping 15 before talking to API
Feb 08 18:30:00 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:00 DEBUG (MainThread) [custom_components.hildebrandglow_dcc.sensor] Update time, sleeping 21 before talking to API
Feb 08 18:30:00 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:00 DEBUG (MainThread) [custom_components.hildebrandglow_dcc.sensor] Update time, sleeping 39 before talking to API
Feb 08 18:30:02 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:02 DEBUG (SyncWorker_1) [custom_components.hildebrandglow_dcc.glow] get 2: (https://api.glowmarkt.com/api/v0-1/resource/XXXX/readings?from=2022-02-08T00:00:00&to=2022-02-08T23:59:59&period=P1D&offset=0&function=sum)
Feb 08 18:30:07 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:07 DEBUG (SyncWorker_0) [custom_components.hildebrandglow_dcc.glow] get 2: (https://api.glowmarkt.com/api/v0-1/resource/XXXX/readings?from=2022-01-01T00:00:00&to=2022-02-08T23:59:59&period=P1Y&offset=0&function=sum)
Feb 08 18:30:10 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.gas_consumption_today is taking over 10 seconds
Feb 08 18:30:10 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.gas_consumption_year is taking over 10 seconds
Feb 08 18:30:10 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.gas_tariff_standing is taking over 10 seconds
Feb 08 18:30:10 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.electric_tariff_standing is taking over 10 seconds
Feb 08 18:30:10 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.gas_cost_today is taking over 10 seconds
Feb 08 18:30:10 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.electric_cost_today is taking over 10 seconds
Feb 08 18:30:15 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:15 DEBUG (SyncWorker_0) [custom_components.hildebrandglow_dcc.glow] get 2: (https://api.glowmarkt.com/api/v0-1/resource/XXXX/tariff)
Feb 08 18:30:21 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:21 DEBUG (SyncWorker_0) [custom_components.hildebrandglow_dcc.glow] get 2: (https://api.glowmarkt.com/api/v0-1/resource/XXXX/readings?from=2022-02-08T00:00:00&to=2022-02-08T23:59:59&period=P1D&offset=0&function=sum)
Feb 08 18:30:22 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:22 DEBUG (SyncWorker_5) [custom_components.hildebrandglow_dcc.glow] get 2: (https://api.glowmarkt.com/api/v0-1/resource/XXXX/readings?from=2022-02-08T00:00:00&to=2022-02-08T23:59:59&period=P1D&offset=0&function=sum)
Feb 08 18:30:30 HA 62d3ea2a38d1[1066]: 2022-02-08 18:30:30 DEBUG (SyncWorker_6) [custom_components.hildebrandglow_dcc.glow] get 2: (https://api.glowmarkt.com/api/v0-1/resource/XXXX/readings?from=2022-01-01T00:00:00&to=2022-02-08T23:59:59&period=P1Y&offset=0&function=sum)

Only query API between 0 and 5, and 30 and 35 Random delay up to 2 minutes when querying Sleep 5 minutes after to make sure we only do 1 update in the window

Narrow window to 2 minutes around each half hour Return interval to every 2 minutes Remove 5 minute wait at end

Fix typo

Remove unnecessary logging when no action required Move comment to correct line

We use asyncio.sleep not sleep

ColinRobbins · 2022-02-08T18:44:21Z

You are seeing the warning in the logs, because the approach of sleeping in the update code works against the HA design principles.
The following would seem a simpler solution to the same problem.
#135

Add required whitspace

jackyaz · 2022-02-08T18:54:18Z

Isn't the polling based on when HA starts the sensor, so if it starts at 12, the poll would be at 12 and 42, rather than around 0 and 30 when the data from Glow updates? I suppose it's not an issue, it just means you could have up to 29 minutes delay in the data (29 and 59 being the worst case)

ColinRobbins · 2022-02-08T19:03:07Z

I am not sure I have seen an thing that says the update must be at 00 and 30 minutes past the hour? Rather called only every 30 minutes.
As far as I know the new data is only polled when glow.py calls the catchup URL.

If it has to at 00 and 30, the I’d suggest we need to modify the code to integrate with HA automation in some way. Adding a sleep in the code works against the HA design principles. Hence the log warning.

jackyaz · 2022-02-08T19:10:50Z

https://forum.glowmarkt.com/index.php?p=/discussion/comment/544/#Comment_544

You need it to send once every 30 minutes, but soon after the actual :00 or :30

ColinRobbins · 2022-02-08T19:42:16Z

If that’s the case, I suggest we need to look at an alternate HA friendly fix. Rather than sleep, maybe return an error of some kind. Then modifying the sensor to reuse old data.

On Tue, 8 Feb 2022 at 19:11, Jack ***@***.***> wrote: https://forum.glowmarkt.com/index.php?p=/discussion/comment/544/#Comment_544 You need it to send once every 30 minutes, but soon after the actual :00 or :30 — Reply to this email directly, view it on GitHub <#136 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGXZ6KN4G4HKTQC2C5C4TTU2FTENANCNFSM5N3JAUWA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

-- Colin Robbins (from mobile)

jackyaz · 2022-02-08T20:01:52Z

Fine by me - I'm new to HA (<1 month since I started using it) and wanted to try and help out fix the addon/integration to keep Glow happy and not turn off the free DCC service :)

ColinRobbins · 2022-02-08T21:00:37Z

Sorry if my comment seems negative - don’t be discouraged, there is a big learning curve on getting started with programming HA, we were all new to it once. A halfway interim fix, might be to set the retry interval to 10 or 15 minutes. That way it’ll only do a few “wasted” retries, but also reduce the time delay past the hour.

On Tue, 8 Feb 2022 at 20:02, Jack ***@***.***> wrote: Fine by me - I'm new to HA (<1 month since I started using it) and wanted to try and help out fix the addon/integration to keep Glow happy and not turn off the free DCC service :) — Reply to this email directly, view it on GitHub <#136 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGXZ6ONTR4UA6AZWPTLAIDU2FZDZANCNFSM5N3JAUWA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

-- Colin Robbins (from mobile)

ColinRobbins · 2022-02-08T21:34:32Z

Thinking about this more, we’ll have to be careful not to hardwire a specific time.
If we hardwire a time, every HA instance using the integration will trigger at the same time - with a potential impact of causing a distributed denial of service attack them.
So we have to make sure there remains some element of randomness to it, within a given time window.

Minimum 1min sleep

jackyaz · 2022-02-08T21:45:13Z

Indeed - which is what my PR does - in the 2 minute window around 0 and 30 it adds a random sleep between 1 minute and 3 minutes for each call - which hopefully will stagger the calls. Though that's with an ugly sleep approach!

jackyaz · 2022-02-08T21:46:19Z

Sorry if my comment seems negative - don’t be discouraged, there is a big learning curve on getting started with programming HA, we were all new to it once. A halfway interim fix, might be to set the retry interval to 10 or 15 minutes. That way it’ll only do a few “wasted” retries, but also reduce the time delay past the hour.
On Tue, 8 Feb 2022 at 20:02, Jack @.> wrote: Fine by me - I'm new to HA (<1 month since I started using it) and wanted to try and help out fix the addon/integration to keep Glow happy and not turn off the free DCC service :) — Reply to this email directly, view it on GitHub <#136 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGXZ6ONTR4UA6AZWPTLAIDU2FZDZANCNFSM5N3JAUWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.Message ID: @.>
-- Colin Robbins (from mobile)

Not at all - my approach isn't ideal but I couldn't easily see a way to have a sensor update run on a schedule, only an interval. I'd need a better understanding of HA and Sensors overall

ColinRobbins · 2022-02-09T08:38:19Z

HI, I am testing an alterative fix to this PR and #135 (@si458)
If it works during the day, I'll make a PR later merging in #80 as well.

I've moved your time test from glow.py to sensor.py, and left the SCAN_INTERVAL at 2 minutes. I have modifed the call to the _glow_update function as follows:

    async def _glow_update(self, func: Callable) -> None:
        """Get updated data from Glow."""
        if self.initialised is True:
            minutes = datetime.now().minute
            if not ((0 <= minutes <= 2) or (30 <= minutes <= 32)):
                # only need to update one per every 30 minutes
                # anything else Glow will ignore
                return

        self.initialised = True
        try:
            self._state = await self.hass.async_add_executor_job(
                func, self.resource["resourceId"]
            )

This way HA will still make a call to the code evey 2 minutes (backing off #135)
If it is outside the time window, the update function will not be called,
HA will effectively randomise when in the 2 minute window the function gets called, based on when HA started and any execution delays.

The self.initialised part is there, so that when HA first starts, it puts a call in no matter where we are in the time window.

It may seem an overhead in calling the fuction every 2 minutes, but the overhead is tiny (and I'd argue it does if now - only worse as it makes the call out to Glow). Its like adding a condition in an automation.

jackyaz · 2022-02-09T08:44:27Z

Makes sense to me!

jackyaz · 2022-02-09T08:46:59Z

I wonder if we should increase the scan interval to say 3 or 4 minutes and widen the window so that API calls can be further spread out?

ColinRobbins · 2022-02-09T09:09:10Z

I did think about making it 5 minutes, but given the current code is 2 minutes kept it as 2.
Not sure - @si458 do you have a view on this?

si458 · 2022-02-09T09:29:43Z

@ColinRobbins sorry been driving to work haha

Personally my PR to change the interval to 30mins has been perfect!

No issues so fair and it's a simple easy fix no messing around with timers etc

HandyHat · 2022-02-22T23:45:17Z

Closing this as a similar solution is in #138. Thanks @jackyaz!

jackyaz added 5 commits February 8, 2022 16:42

Update sensor.py

9bb03f4

Only query API between 0 and 5, and 30 and 35 Random delay up to 2 minutes when querying Sleep 5 minutes after to make sure we only do 1 update in the window

Update sensor.py

4baa43a

Narrow window to 2 minutes around each half hour Return interval to every 2 minutes Remove 5 minute wait at end

Update sensor.py

18414f0

Fix typo

Update sensor.py

c86d8f6

Remove unnecessary logging when no action required Move comment to correct line

Remove unused import

d146a03

We use asyncio.sleep not sleep

Update sensor.py

18467b6

Add required whitspace

Update sensor.py

0f62060

Minimum 1min sleep

This was referenced Feb 9, 2022

Issue / Bug Roll Up #137

Closed

Issue / Bug Roll Up #138

Merged

HandyHat mentioned this pull request Feb 9, 2022

No longer working? #126

Closed

HandyHat closed this Feb 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update sensor to only query Glow API shortly after 0 and 30 mins past the hour #136

Update sensor to only query Glow API shortly after 0 and 30 mins past the hour #136

jackyaz commented Feb 8, 2022 •

edited

ColinRobbins commented Feb 8, 2022 •

edited

jackyaz commented Feb 8, 2022 •

edited

ColinRobbins commented Feb 8, 2022

jackyaz commented Feb 8, 2022

ColinRobbins commented Feb 8, 2022 via email

jackyaz commented Feb 8, 2022 •

edited

ColinRobbins commented Feb 8, 2022 via email

ColinRobbins commented Feb 8, 2022

jackyaz commented Feb 8, 2022

jackyaz commented Feb 8, 2022

ColinRobbins commented Feb 9, 2022 •

edited

jackyaz commented Feb 9, 2022

jackyaz commented Feb 9, 2022

ColinRobbins commented Feb 9, 2022 •

edited

si458 commented Feb 9, 2022 •

edited

HandyHat commented Feb 22, 2022

Update sensor to only query Glow API shortly after 0 and 30 mins past the hour #136

Update sensor to only query Glow API shortly after 0 and 30 mins past the hour #136

Conversation

jackyaz commented Feb 8, 2022 • edited

ColinRobbins commented Feb 8, 2022 • edited

jackyaz commented Feb 8, 2022 • edited

ColinRobbins commented Feb 8, 2022

jackyaz commented Feb 8, 2022

ColinRobbins commented Feb 8, 2022 via email

jackyaz commented Feb 8, 2022 • edited

ColinRobbins commented Feb 8, 2022 via email

ColinRobbins commented Feb 8, 2022

jackyaz commented Feb 8, 2022

jackyaz commented Feb 8, 2022

ColinRobbins commented Feb 9, 2022 • edited

jackyaz commented Feb 9, 2022

jackyaz commented Feb 9, 2022

ColinRobbins commented Feb 9, 2022 • edited

si458 commented Feb 9, 2022 • edited

HandyHat commented Feb 22, 2022

jackyaz commented Feb 8, 2022 •

edited

ColinRobbins commented Feb 8, 2022 •

edited

jackyaz commented Feb 8, 2022 •

edited

jackyaz commented Feb 8, 2022 •

edited

ColinRobbins commented Feb 9, 2022 •

edited

ColinRobbins commented Feb 9, 2022 •

edited

si458 commented Feb 9, 2022 •

edited