Performance improvement - Entity filter proxy

[davidcoulson]

I stumbled across this repo earlier today: https://github.com/fuslwusl/homeassistant-entity-filter-proxy

Basically does the same thing a remote HA instance does in limiting entity state updates, but seems to do it automatically with less configuration required.

I've not been able to get the add on to work but it does work great if I just run it from command line on another system.

I don't have any good benchmarks yet - will try to pull something together tomorrow.

[davidcoulson]

Actually this one is much easier to setup using the addon.

https://github.com/GabrielGoldsteinAnidea/HA-Websocket-Stripper

[maxlyth]

These both look interesting and I was considering writing something similar myself so I might pick these apart out of curiosity however I should point out that another large part of my secondary HA is pre-rendering dashboards. I have multiple wall panels and their dashboards are similar but obviously different for each room.

At first I tried defining my lovelace dashboards is yaml and using gomplate to render room customisations but the HA foundation has made a decisive turn to JSON dashboards and nobody wants to edit dashboards outside the UI so I switched to Streamline which is awesome but the templates are expanded and rendered at point of paint rather than point of editing which puts the entire load on the rendering device.

Therefore while the websocket entity firehose is usually the reason a panel feels "dead" to control buttons due to state update backlog, the time to render a dashboard is not fixed by pruning the entity set. I know the Streamline project have improved the situation in more recent releases but I still get better performance by pre-rendering my templates using homeassistant-lovelace-to-yaml.

[davidcoulson]

I have over 20k entities in my HA environment, so cutting down the updates to <200 really helps a lot with performance vs all the other changes.

I can't figure out how homeassistant-lovelace-to-yaml is supposed to make anything faster? Have you done any A/B testing with the different capabilities to figure out what has the most back for the buck/quid?

[davidcoulson]

FYI I did some quick testing using HA WebSocket Stripper on my Android 14 tablet.

Using my full dashboard (which pulls in ~200 entities) and loading it via Fully Kiosk

• Native - 32s
• Stripped - 9s

I did basic checks to make sure functionality was as expected, but this clearly had a massive improvement on user experience.

[maxlyth]

I have made some significant progress on a one stop solution to this problem. One of the benefits I gain from creating the built-in webview in the 0.9 release was direct access to the Chromium webview which in turn enabled a very important trick.

Normally an HA dashboard subscribes to every entity state change on the HA server and that is really important to maintain dashboard coherence for general purpose interaction. However dashbaords are used for a very limited set of the full HA user interface and over 90% of the entities on a typical system have zero purpose for showing state and controlling devices.

The trick is that I can intercept the outbound entity subscription request from the panel and replace it with a tiny subset of entities that are useful for dashboard rendering/control. In my home I had previously solved this with a very complex system of two HA servers where one subscribed to a tiny subset of entities and served the dashboards. This worked for me but is not simple to setup.

I have already tested a proof of concept version that renders HA dashboards direct from the full server as fast as my hand tuned server and far faster and responsive than the default. This effectively makes solutions like HA WebSocket Stripper redundant and while I have not yet designed a UI to configure this, I'm sure I can do a better job than other products out there and because it is client side there is no messing around with certificates to get full https working.

In truth this was the capability I hoped for when I started this project but I had to build everything so far to test whether it would actually work in practice. There could still be a blocker I can't solve but I am already working on a preview UI (maybe 0.9.3) so I can get this out for other people to test.

I am hopeful that this is the feature that is worthy of a v1.0 release.

[davidcoulson]

I am hopeful that this is the feature that is worthy of a v1.0 release.

100% a 'killer feature' that would drive ha-paneld use over HA native app or other kiosk tools. Sounds perfect for a 1.0 :)

For configuration I'm assuming you could just grab the dashboard in the background without rendering it and figure out what entities states are needed. Benefit over the proxy tools is you could refetch periodically and add new entities if the dashboard changes (I have to cycle the HA WebSocket Stripper docker right now to do this). I'm yet to find a reason to explicitly block any entities, but I do have a few that I say "I always want these" dashboard scrape + override is probably 99% of use cases.

I'll keep my eyes open for a build with this and can compare it to the other approaches I'm using today. Would be super cool to have some statistics, like you have X entities and we're using Y, or being able to do a background render and compare performance to demonstrate the value of the optimization. Last I checked I was using around 160 of 22k entities for my dashboard, so upside is pretty ridiculous.

[maxlyth]

It's coming. I am having to do some extra work to guard against overly broad expansion of auto-entities type dashboard extensions from creating firehose entity subscriptions.

Currently in the UI you can choose whether to automatically promote an entity use to a subscription but what worked for me was to leave that off and just browse through all the tabs on the dashboard and then pin all the entities it discovered for future use.

This got my average entity counts on all mu dashboard to around 50 (down from over 3000). I also show stats for each entity so you can see the ones that are flapping on HA and might need refactoring.

[maxlyth]

Finally 0.9.2 has landed with its built in entity filter. The new feature is self-learning for each panel and makes redundant the need to deploy a centralised entity filter proxy.

[davidcoulson]

Btw I was trying to get some benchmarks of filters vs not filtered. When filtering was disabled it ran 100% cpu and never updated render time.

Would be nice if there was a way to programmatically get performance data with filtering disabled.

[maxlyth]

I know that on my own system with ~4k entities that a quad core Arm Cortex-A35 runs at 100% and emits a stream of websocket state overflow errors from the chromium renderer. This was why I originally set up my dual HA server boondoggle. Only my iPhone and a brand new Octo-core wall panel could connect to the main HA server without being overwhelmed. With the entity filter I have all my panels now connecting to the same, single HA server and they work better than my old dual server setup.

I am sure there are still going to be some kinks to fix in the entity filter but if you have a mature HA with more than 1k entities this feature is essential if you are going to run wall panels, I'm proud of my work, and I think it is the only product worth anything in the market. The next 0.9.5 release is also huge, especially for anyone with security concerns and I have been working night and day to get it ready.

The project is still not v1.0 finalised but I am hoping that it gets some publicity and a wider audience soon for early feedback.

[maxlyth]

The entity filter is still far from perfect and still requires ha-paneld to inspect and parse the dashboard for configuration that only resolves to an actual entity list at runtime. Classic examples of this are the auto-entities custom integration. In 0.9.6 I am expanding support to a few more common integrations but this will never be complete and the code will always flag a warning if a wildcard expands to a large entity list as that is likely a bad dashboard design choice for a wall panel.

I bricked one of my panels today and had to set it up from scratch and had to go through the entire entity filter setup process from the perspective of a new ha-paneld user and it was far from perfect. It is a grind from a testing perspective but I am planning to reset one of my panels multiple times before the release of 0.9.6 so I can improve the current onboarding experience that I admit is currently really clunky for new users