Configurable scan interval for integrations

[farmio]

Describe your core improvement

It would be great if integrations were allowed offer custom scan intervals. For some integrations it may fit to offer modes in form of a select selector / select entity (like high and low) and for others it may be more appropriate to offer a duration selector / number entity.

Current limitations

Currently integrations are not allowed to offer user configurable scan intervals. The canonical way to modify the scan interval is to globally (for an integration) disable polling and create an automation to do it manually - which is imho not very user friendly and may lead to unobvious issues - eg. when the integration uses multiple coordinators

https://www.home-assistant.io/common-tasks/general/#defining-a-custom-polling-interval
or
https://community.home-assistant.io/t/how-to-set-a-custom-scan-interval/680579

Technical benefits

Users would have a nice way to configure their update intervals to their likings from a config flow, options flow or a configuration entity.
A configuration entity would be very easy to automate to eg. have slower intervals after sunset and faster after sunrise without a user needing to know integration internals.

Offering sane options for integrations would prevent users to go the manual route and configure too low intervals that may run into rate limitations of APIs while still being able to provide integration-specific dynamic intervals (eg. for solar equipment slower at night).

Additional context

No response

[joostlek]

So far we only allow it via this automation. Mostly because the automations give the user way more power. Poll whenever you like, without doing code changes.

Otherwise we would have a slippery slope of people wanting to update with special cases

[Petro31]

I think the point of @farmio's post was to have an easy way to change simple polling frequencies, while keeping the current automation method for anything outside that. It's not an outlandish request either and it should be considered in my opinion. The system options page is quite small and could easily hold additional properties for simple scan intervals.

[frenck]

We can keep it around as a feature request. However, architectural discussions are backing this one up; likely not to change any time soon for that reason.

[sveinse]

A small question on the side: Are scan intervals centrally defined (by policy), or are they integration specific?

Different cloud counterparts have different policies when its comes to polling frequencies, so I doubt there can be one global setting. Moreso, the allowable polling frequency might be dependent on state or time (e.g. less frequent polling at night, more frequent polling when charging, etc.). Even different users might have different agreements with cloud provider that affects the scan intervals.

I think the integrations that have use-cases like this should be able to offer the ability to customize the scan intervals. That doesn't mean that all integrations needs this, thou. The means to turn off automatic polling and use automation still stands. This is great freedom for the user.

[frenck]

are they integration specific?

They are integration-specific; some are even dynamic based on the number of accounts or resources configured.

I think the integrations that have use-cases like this should be able to offer the ability to customize the scan intervals.

We had that, and forbid it over time. The reason is that tons of integrations cam up with lots of configuration options. E.g., some dynamic elements, time based polling (e.g. reducing usage during the night), and even some with flexible conditions (e.g., don't poll when not home).

The problem was: The amount of configuration options, differences between integrations, and not even starting about the abilities to let people shoot themselves in the foot, resulted in no longer allowing scan intervals to be set.

Instead, we now provide sane defaults out of the box. If someone wants to go advanced, and automation can be used; in which one can build any logic, schedule or interval, build around any condition, one would like or can even imagine.

../Frenck

[sveinse]

Understood. I think this gives two alternatives:

1. Integrations needs to be more complex to cover the span of use cases automatically
2. Let the user deal with it. Especially the corner-cases.

I'm here because of the 3rd party integration I'm maintaining (Zaptec). The cloud vendor has reached out to us because we're polling too frequently and "deeply" in accordance with their API policy. They have a concept of "shallow" poll and a "deep" poll with different scan intervals that must be addressed in the integration. The problem is that the user base is diverse in how many devices they have and how often they can poll. What's good for one user, might rate-limit for others. What's fast enough data updates for one is too slow for another.

TL:DR: I need to make a more elaborate scheme to deal with the various types of poll and intervals that also considers the number of devices. This takes considerable time and efforts to make and I'm even not sure how I might verify that the algorithm I devise is sufficient for the users. That's why I was thinking that exposing these parameters to the user configuration can be a fix for those special cases where it isn't sufficient.

If the user implements a custom scan interval, how does that work if there are integrations with different "layers" of intervals, such as "shallow" poll and "deep" poll example from above?

[joostlek]

You can for example split the coordinator into multiple ones.

How is a user differentiated on how many times they can poll? Because if it's just the number of devices, you can use that information from the API to make the poll interval depending on the amount of devices someone has

[sveinse]

Yeah, its more complicated. It depends on the user (some users seems to be able to poll more often), the state of the device (charging or not), number of devices. All in all the integration needs to do some elaborate book-keeping on polling.

This is a bit OT to the TS topic, but I'm taking the liberty:

How does the integration get notified on homeassistant.update_entity actions and its argument? In the case of having multiple coordinators and/or dynamic poll intervals, the integration (i.e. the coordinator) needs to know what has been triggered. DataUpdateCoordinator._async_update_data() accepts no arguments.

[gjohansson-ST]

With that action you tell the entity to update which in turn is asking the coordinator to update which it is "attached" to. It does not ask all the coordinators for the integration so it's also important to target the correct entity in the action depending on what you want to have updated.
For complex integrations with multiple coordinators, that might not be entirely obvious to a user.

[sveinse]

Another consideration is that if users chose to disable polling to an integration, I assume it will disable polling for all coordinators that the integration use. If the integration uses multiple coordinators, the user will need to take care of updating all of them manually. homeassistant.update_entity is triggering updates for one specific entity which can be connected to one coordinator. But this might not cover all entities of the integration which the automatic update handled. For the case of multiple coordinators, this means that the user need to trigger multiple entities to get everything updated.

I think this guide https://www.home-assistant.io/common-tasks/general/#defining-a-custom-polling-interval seems to imply that all its entities gets updated on update_entity, which might not be the case. This will be confusing for the user if the integration happen to have multiple coordinators and will observe partial updates.