Add support for logging

[rvk]

It would be useful to be able to see long term history in group monitor. There should be a preference for retention period and a cut-off value for realtime browsing, for when your sort the group monitor according to criteria other than time.

In addition to group monitor there could be object view like Schneider Wiser for KNX, where you can browse/filter group addresses and can see the last value seen on bus for that group address and when it was seen.

You're doing magnificent work with the integration.

[farmio]

Hi 👋!
Regarding long term history: Have you seen #92 ?

You can "filter" for group addresses with the search bar. It's not really the most handy way to solve that, but for general debug purposes it should be good enough imho.

This object view sounds interesting. Maybe it can be combined with #76 one day 🤔

[rvk]

I commented on #92 and clarified the use case and possible solution there.

The current filtering is no worse than what's in ETS, so everyone is accustomed to it anyway :)

The cut-off time limit for realtime browsing would be useful to avoid loading entire, possibly very long history in memory and on page.

And yes, such object view would also solve #76 .

[farmio]

current max history is 5000 Telegrams - that's about 2MB of data. I don't think this would lead to problems regarding loading times or responsiveness, but it is indeed not very handy to have it on one page.
Maybe we can add pagination to the group monitor somehow... but I'm not sure how to handle that with incoming live data 😬

[KomtGoed]

Hope this is the right "forum" to discuss, if not, please redirect me.

Have been moving away from my current KNX platform during the last half year and am very pleased with the combination HA+NR, thanks!!!

The last feature for finally getting rid of the old platform is long-term logging and easy searchability of historical data (AKA extensions of the group monitor):

• Log ALL KNX messages to (I guess) InfluxDB, with everything contained in the telegrams plus the dpts available in HA.
• Easy user interface where one can add multiple filters to get a narrowed-down list of telegrams for analyzing: data and time periods, source, destination
• Filters can be saved and are clickable for en-/disabling
• This implies a way to dis-/enable realtime monitoring.

My first impression is that the out-of-the-box InfluxDB integration is too limited, e.g. because it does not support multiple targets as far as I understood. Was looking into knx2influx, but that's too limited in terms of functionality at this point in time and seems abandoned. Plus totally unclear how to build a good UI... don't think grafana would be suitable.

Given my limited programming experience, I would rather support a community effort with testing etc. than trying to build something on my own.

How does the above sound to you?

[farmio]

Hi 👋!

Having DB-connectors in the HA KNX integration would be out of scope imho.
I don't think logging to databases like influxdb should be a concern of the HA KNX integration. This could probably be done by a HA automation using knx_event or knx.telegram trigger (once it is merged) or the interface device trigger. Otherwise I guess there are better tools out there for exactly this purpose (Telegraf etc.).

For the better UI, I am 100% with you. Since HA data table was revised recently (2024.4), it may be possible to leverage its new filter methods for the group monitor.

Saving filter options sounds like a lot of work for a feature that is quite rarely used. For me the HA group monitor is a tool to quickly test things or look up addresses - it was not meant to be a full fledged KNX logging and debugging tool. That said, if anyone cares to contribute to push it that direction, I'm fine with it too 😉

[KomtGoed]

Hi, thanks for the quick response.

Sounds indeed logical to move the DB integration higher up the chain, HA automation makes sense. As long as it is able to capture all KNX traffic even if the GA with DPT has not been pre-defined. For me, that is the argument against Telegraf because that integration only captures previously defined GAs. (I know, I know, in a well-defined KNX environment, all DPTs should be known upfront. But reality is harsher I guess ;) And the need to keep exporting/importing the ETS project file is a bit of a hassle.)

Do I understand you correctly that if someone comes with an "extension" to knx-frontend that integrates with InfluxDB for reading "telegrams" from there, that would be OK? Or would this rather be something for HACS?
In any case, seems like another (set of) rainy weekend(s) is needed for studying https://developers.home-assistant.io/docs/frontend/ Or finding supporters...

[farmio]

Hm... you need to import ETS project files for HA as well. Maybe have a look here https://github.com/svsool/knx-telegraf-config-generator

Anyway I'm not sure if an influx connector would work solely from the frontend - without any backend-part.
How many telegrams would you plan to store there? Days, months or years?

[KomtGoed]

Thanks for your thinking along and patience.
Would be at least a year with ability for more. That's what I am used to from the current KNX platform... (and has proven good to have in those rare circumstances)

I see your point on the backend (shows my level of experience :()...

From our earlier chat (which I just remembered, https://community.home-assistant.io/t/knx-cookbook/230972/447), a "catch all" would work as well without importing the ETS project file (which of course I do for 99% of the installation)?

Question might come down to: Is a KNX group (and potentially bus) monitor with historical data based on HA a feature more HA users would want/need? Or is that something more for e.g. system integrators (likely with the related price tag)?
I am afraid that the learning curve would be a bit too steep for me the coming months to tackle this on my own.

[farmio]

You can't build a bus monitor with a tunnelling data link layer connection - only a group monitor.
I don't know how many users would need this feature - can't think of a situation where I would like to find a specific telegram that was sent on the bus 2 months ago. But your milage may vary.

[KomtGoed]

Thanks for the clarification, I'll leave the bus monitor to the old platform then (with built-in KNX interface).
Regarding the group monitor: mileage varies indeed, I've had these situations. And even for debugging for the current and past few days, it's an absolutely brilliant feature: Why was a certain light suddenly turned on? Was that from the visu? Or through a motion detector? Or from an NR flow? etc. etc. Once you have the feature, the use cases are numerous.

I'll put it on my to-do list for the coming months/year.

[farmio]

To get the past few days, you can already use HAs Group monitor. Just set the retained telegrams to a higher count in the Knx integration settings.