Reduce Home Assistant DB Updates

[rct]

This integration generates a lot of data. I've got 170 entities for my modem. This could cause a lot of growth to the Hass DB over time and writes to the Hass system disk.

Let's start with low hanging fruit: The System Uptime entity generates an update on every polling cycle. This is somewhat redundant, if the Last Boot Time entity is available, system uptime can be displayed from that (now - boot time). The Activity card on the device page is filled with the updates to system uptime, so if other values changed, it is hard to see.

Since 3.14, which is still in beta, is going to be a major release for this integration, it might be a good time to remove that entity.

[kwschulz]

Thanks @rct, good thread to open.

You're right that System Uptime is noise but I think most find it valuable on their dashboard so I haven't given it much thought. However, we might be able to leverage the Dashboard generator to derive the value and just persist the last boot time sensor, which we derive internally in most cases anyway. The wrinkle is in the current code: Last Boot Time is computed as now minus uptime with no rounding, so it drifts a few seconds each poll and writes every cycle too. That drift is a problem to solve so the timestamp only changes on a real reboot.

On the database side, uptime is one row among many. The per channel signal sensors (power and SNR especially) change on nearly every poll, so on a 32 channel modem that's well over a hundred rows written every cycle, with the error counters on top of that. That part is bounded though, since the recorder purges it after the default 10 days. The longer term growth is the aggregate history: every sensor with a state class logs long term statistics on a fixed schedule whether the value changed or not, and that's kept for years. That scales with the number of entities, which is really why being able to switch off groups would help.

The broader question I'd rather put to you and anyone else watching: would it be more useful to switch off the noisy high cardinality groups (the per channel error counters especially) than to remove any single entity? And if so, would you rather toggle whole groups at setup, or have them created but disabled so you turn on just the ones you want?

If we go this route it'd be a community discussed enhancement, so this is the right place to hash it out. Curious what fits how people actually use it.

[rct]

The broader question I'd rather put to you and anyone else watching: would it be more useful to switch off the noisy high cardinality groups (the per channel error counters especially) than to remove any single entity?

Yes, but I'll talk about that later in terms of use cases. I brought up the system uptime entity because it seems like an easy source of noise (and wear and tear) to remove for data that can be gotten from elsewhere. I can't see any other activity on the device page because it is filled with system uptime updates. (See below).

(Side Note: I find Home Assistant's Device Pages to be a very useful tool, but it depends on how well organized the device's entities are. More on that later.)

Last Boot Time is computed as now minus uptime with no rounding, so it drifts a few seconds each poll and writes every cycle too.

Even without rounding, I'm seeing a handful of updates in 24 hours. (See below). And as you indicated rounding from second resolution to minute resolution (or even 2 minutes) would probably remove all of the jitter.

I could probably do without both, since that data is already stored as the last change time on Docsis Status or System Status. Unfortunately it does get changed when Home Assistant restarts.

BTW, This seems to be a common pattern with some new integrations that start out by storing things like uptime/connected time etc in a noisy way.

[kwschulz]

Thanks for this, and I see your point. I'm working down the open issues right now, but reducing the recorder and entity load is squarely on my list before I officially release v3.14, and I'd value your input as I shape that. My goal is to keep the project focused on capturing what the modem reports with minimal normalization, but as I've learned, every manufacturer has its own quirks. This is my first HA integration, so I'm learning as I go. Really appreciate you helping make it better.

[kwschulz]

Again, thank you for catching this @rct! Latest beta is significantly less "noisy". System Uptime and Current Time (this should have never been a sensor) are gone, Last Boot Time is one stable timestamp, and an outage now writes a single Activity line instead of four because the slow-moving sensors hold their value through a blip. The standalone Docsis Status sensor is folded into Status as well.

I value your experience with HA integrations, especially since this is the first one I've attempted and have learned a ton along the way. So please check out beta.14 and let me know if there are other optimizations you see.