always_callback does not trigger a state update within homeassistant

[CanisLupus-42]

Description of problem:

A KNX sensor is defined with always_callback, but does not trigger a state update within homeassistant based on any GroupValue_Response. Even if it is a direct response to a GroupValue_Read triggered bei the sync_state of the sensor itself.

The green GroupValue_Write are found in the state db in homeassistant (screenshot form the SQL db, time mismatch by 3h due to timezone), and do show up in the developer-tools/event tab, when listening to state_changed. Even, when the same value is sent twice, as it is expected for always_callback.

GroupValue_Response don't show up anywhere, therefore leading to the paradoxical, that a sensor is neither changed, nor updated for many minutes, even it is set to sync_state: expire 1 and always_callback: true

• using xknx standalone
• using Home-Assistant knx integration

Version information:

• xknx / Home-Assistant release with the issue: 0.21.3
• last working xknx / Home-Assistant release (if known): -

KNX installation:

It does not seem hardware related, can be reproduced within HA, nevertheless also occures also with real hardware KO connected to GAs.

Problem-relevant configuration.yaml entries (fill out even if it seems unimportant):

knx:
  sensor:
    - name: knx_test_input
      state_address: "0/0/255"
      type: percent
      sync_state: expire 1
      always_callback: true

  expose:
    - entity_id: input_number.knx_test_output
      type: percent
      address: "0/0/255"

input_number:
  knx_test_output:
    name: knx_test_output
    initial: 50
    min: 0
    max: 100
    step: 1

Diagnostic data of the config entry (only when Home Assistant is used)

Traceback (if applicable):

[farmio]

I'm curious what the use case for that would be.

Always_callback was initially meant for devices sending periodically on their own or sending things like scene numbers that are meant to trigger events even with same payloads (although knx_event would be more appropriate).

A GroupValueResponse would always trigger a state change event when there is a new (changed) state.
Changing this behaviour would certainly break some automations that are not aware of what sync_state even does.

[CanisLupus-42]

The main point I stumbled on is the visualization in grafana using influxdb as a datasource, which relies on the state_update to generate a new datapoint. If a value is not sent regularly by a KNX device, there are no datapoints to be displayed in certain timeranges. The fill(previous) parameter does not help either, as it requires at least one datapoint to reference off in the selected timerange. This leads to empty or half empty graphs for rarely changing entities.

[farmio]

This sounds more like an InfluxDB issue then 🙃

[CanisLupus-42]

Probably you are right. When looking around the internet, I found some other people in e.g. the knx forum with the same 'problem' for years, therefore I thougt it might be handy. Some more or less hacky workarounds seem to do the trick, but somtimes with side effects like heavy load on the database.
As it is apparently not a bug, but a feature, I would raise the question: Is there any interest in a feature request for this? Maybe add another option to explicitly allow state_updates based on GroupValue_Responses? Or is this out of scope for this library? If there is any interest I would try myself at adding this feature, even python is not my main language (which is probably apparent).

[farmio]

Just found this and well, it seems to be a known problem for quite some time. 😦
influxdata/influxdb#6878

[CanisLupus-42]

Yep. It seems to be in the implicit category 'Won't fix' ...
Could (or should) this issue be addressed in xknx? As mentioned possibly as a new option (default off, so not breaking any existing deployments) Or is it a niche application far from the core functionality of this library, that it does not belong here?

[farmio]

I think the influx issue should be specifically handled in the influxdb integration - by some expiration timer or such writing to the db regularly. Its not a KNX issue as this happens to any other integration too...

For the new option, I currently don't see any other usecase than bypassing this influx issue 🤔 has @marvin-w an opinion about this?

[CanisLupus-42]

@farmio I (now) totally aggree with you. This is a general problem in HA or the InfluxDB integration.

I checked my other integrations (ESPHome, MQTT) and saw that I implemented the equivalent of always_callback (force_update) everywhere and just forgot about it. Therefore I thought KNX would be the cause for the problem.

Funny enough: The MQTT documentation explicitly states, that this is the way to go for this scenario. 😬

force_update boolean (Optional, default: false)
Sends update events even if the value hasn’t changed. Useful if you want to have meaningful value graphs in history.

So there might be a point for adding the feature to xknx, to be feature equivialent to other integrations? 🤔

I now mitigated the problem by upgrading to InfluxDB 2.2 where queries can be written in Flux rather than in the limited influxql. This is probably the better fix, then trying to work around a HA problem / shortcoming in every single integration. If the feature seems therefore irrelevant or unnecessary, you can simply close this issue.

Just for reference for someone in the future, faceing a similar problem: Here is a flux query that pieces together all values from the selected timerange, the last value before the timerange and a dummy value to continue the graph until the end of the timerange. Now you have a set of datapoints that can simply be displayed in Grafana, with no issues due to missing values in the selected timerange.

bucket = "homeassistant"
measurement = "%"
filter_function = (r) => r["entity_id"] == "knx_test_input"

// Data in the selected timerange
main_data = from(bucket: bucket)
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == measurement)
  |> filter(fn: (r) => r["_field"] == "value")
  |> filter(fn: filter_function)

// Last value before selected timerange
last_before = from(bucket: bucket)
  |> range(start: 0,stop: v.timeRangeStart) // From unix epoch 0 (1970-01-01T00:00:00) until start of the timerange
  |> filter(fn: (r) => r["_measurement"] == measurement)
  |> filter(fn: (r) => r["_field"] == "value")
  |> filter(fn: filter_function)
  |> last()
  |> duplicate(column: "_stop", as: "_time") // rewrite the timestamp to v.timeRangeStart

// Table with all 'real' data
data = union(tables: [main_data, last_before])
  |> window(every: inf)

// Dummy entry for the end of the timerange (continues the line to the end, based on the last value)
end_dummy = data
  |> last()
  |> duplicate(column: "_stop", as: "_time") // rewrite the timestamp to v.timeRangeStart

union(tables: [data, end_dummy])
  |> window(every: inf)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Please make sure to update to the latest version of xknx (or Home Assistant) and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.