Riemann sum integral sensor not updating while source sensor value is stable

[itssimon]

The problem

The Riemann sum integral sensor currently only updates when the source sensor updates, which is not always sufficient. Ideally it would calculate a new value periodically too, even if the source sensor value remains stable.

For example: I’m monitoring the energy used by our pool filtration system. It’s power usage stays exactly the same throughout the day after a brief priming phase. In this scenario the integration sensor does not calculate the energy used until the end of the day when the pump stops.

What version of Home Assistant Core has the issue?

2023.2.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Integration - Riemann sum integral

Link to integration documentation on our website

https://www.home-assistant.io/integrations/integration/

Diagnostics information

No response

Example YAML snippet

sensor:
  - platform: integration
    source: sensor.pool_filter_pump_power
    name: Pool Filter Pump Energy
    unit_prefix: k
    round: 2
    method: left

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant]

Hey there @dgomes, mind taking a look at this issue as it has been labeled with an integration (integration) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of integration can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign integration Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

integration documentation
integration source
_{^{(message by IssueLinks)}}

[rgb-Argentum]

I'm afraid that this is not a 'feature' since integration is a function of values and time. So excluding time completely and keeping the same value until the source sensor value changes means that this integration is only half-implemented. I've tried forcing update of the base sensor and the integral itself through an automation but it didn't change anything. The only 'solution' is to insert some fakes into the sensor value stream time to time. But that makes the original sensor history look ugly.

[dgomes]

The current implementation is event driven to support sensors that update often (which is also when Riemann Sum best approaches the integral function). You are requesting a new feature because it requires a completely new implementation that is based on periodic polling of the source sensor. It's not that it can't be done, it's just that it isn't a bug...

[rgb-Argentum]

I know that the current implementation is based only on sensor update. But that is not exactly true: HASS doesn't broadcast any updates if the new value of the sensor is the same as the previous. That means the current implementation is practically changes only when the sensor value changes and not when the updates received from the sensor device. So the integration should also include an (optional, configurable) timer that periodically updates the integration value based on the last value and the time elapsed since that moment. There is no need to check or poll the sensor.

[kaijk]

My rain gauge sends a new reading every 10 minutes with 0.01 inch resolution. When its raining, successive values are often the same: 0.01, 0.01, 0.02,. 0.02, etc. This wreaked havoc with HA. It missed all the new, valid, values that happened to be the same value as the last value. Each new value is cumulative, not a repeat of the current state if unchanged from the prior. I solved this by adding a small +/- random lambda value to each incoming data point.

This doesn't guarantee that two identical 10 minute readings will not also get identical random adders, but, good enough for me. So, until another solution arises, here's mine.

The command assures that the new data are no more than a minute old so stale data does not get interpreted as new.

In the if else block. I add both positive and negative lambdas to even it out around zero over time, and never add a lambda when the new reading is zero:

- platform: command_line
  name: 'Acurite 10-Min Rainfall'
  command: "[[ $((`date +%s` - `date +%s -r '/config/externaldata/acuriteweatherdata.txt'`)) -le 60 ]] && cat /config/externaldata/acuriteweatherdata.txt || echo '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.0' "
  # Add a small lambda to try to keep successive values unique so the Utility Meter will add each value to the sum.
  value_template:  >-
    {% set val = ( value.split( ',' )[14] | float(0.0) ) | round(2) %}
    {% if val == 0.0 %}
       0.0
    {% else %}
       {{ ( val  + ( [ 0.0, -0.000001, 0.000001, -0.000002, 0.000002, -0.000003, 0.000003, -0.000004, 0.000004, -0.000005, 0.000005, -0.000006, 0.000006, -0.000007, 0.000007, -0.000008, 0.000008, -0.000009, 0.000009, -0.000010, 0.000010,-0.000011, 0.000011, -0.000012, 0.000012, -0.000013, 0.000013, -0.000014, 0.000014, -0.000015, 0.000015  ] | random ) ) | round(6)  }}
    {% endif %}
  unit_of_measurement: 'in'

[dgomes]

To possibly address the lack of updates in some sources, I've created a Helper integration: https://github.com/dgomes/ha_sampler

If you run the source through this custom helper, and then use this custom helper as the source of Riemann Integration, you should get better results.

PLEASE: don't create issues or reply to this thread about this custom_component, use the GitHub repository of the custom_component!

[thecem]

So what is the solution to fix this? to add a helper for a helper?

[dgomes]

The sampler helper is a stop gap for some use-cases.

I don't even know the real impact.

If any of you is testing/using, feedback would be nice.

[kaijk]

I don't see ha_sampler in either Integrations or HACS. I'm on 2023.4, though, if it was included later

…

On Mon, Jun 5, 2023, 12:00 PM Diogo Gomes ***@***.***> wrote: The *sampler* helper is a stop gap for some use-cases. I don't even know the real impact. If any of you is testing/using, feedback would be nice. — Reply to this email directly, view it on GitHub <#88940 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL2THYV4SDXVL36S2OKECNLXJYUFZANCNFSM6AAAAAAVLL4TF4> . You are receiving this because you commented.Message ID: ***@***.***>

[dgomes]

it's a trial, it's not part of HACS (although you can use HACS to install it)

[asimar75]

Work around, suggest to create automation

alias: reload INTEG
description: reload INTEG
trigger:

• platform: time_pattern
  minutes: "1"
  condition:
• condition: state
  entity_id: sensor.ABC
  state: unavailable
• condition: or
  conditions: []
• condition: state
  entity_id: sensor.XYZ
  state: unavailable
  action:
• service: homeassistant.reload_config_entry
  data: {}
  target:
  entity_id:
  - sensor.ABC
  - sensor.XYZ
  mode: single

[kaijk]

@dgomes I'm out for the next month or so and have not been able to test your idea. Sorry, and I really appreciate your digging into this issue.

HA's assuming successive identical sensor readings reflect no-change in the state is a large and lurking problem.

Rain Gauges are a great test case in which successive 10-minute intervals each report, say, 0.01 in of new rain. HA without a fix or workaround ignores those successive identical new incremental readings by assuming the readings are not incremental, but cumulative to date.

[erkr]

My two cents, read the values at least ones per unit_time. Just add a simple timer that will give an additional event when unit_time expired.
I predict that having these extra events won’t harm and don’t require a completely new implementation.
I suspect it will even simplify your current design! Now you somehow have to chop integrations results over various unit time buckets, when the next sensor event took longer than the unit time.

Note: the derivative integration has a very similar issue. When the sensor stops changing, the derivative shall become zero, but doesn't without further sensor updates

[Steve2017]

This looks like it could be the same problem I have with Riemann sum sensors which become unavailable when a source sensor becomes unavailable for a short period while it is producing unchanged data.

I have the Enphase integration that supplies data about electricity consumption and solar production. For some unknown reason the device goes off-line for a minute at 23:00, leaving all consumption and production sensors unavailable for that minute. At that time the solar production is zero and has been for several hours.

Template sensors using the integration sensors (Solar Export Power & Grid Import Power) come back after showing unavailable for the same minute as the integration sensors. This is the solar export power sensor:

        state: >
          {{ [0, (states('sensor.envoy_current_power_production') | int - 
                states('sensor.envoy_current_power_consumption') | int)  /1000] | max }}

The Grid Import sensor resumes showing my consumption as expected (about 500-watts). At this time of night, the solar export sensor is showing 0kW before and after the event, but is available before and after the outage.

Downstream from that I have a Solar Export Energy sensor using Reimann Sum:

  - platform: integration
    name: Solar Export Energy
    source: sensor.solar_export_power 
    unit_time: h
    method: left

This sensor and all sensors relying on it as a source (daily energy export, $ earned from the export, net cost of power etc.) become unavailable and remain that way until solar production resumes the next morning. Even though sensor.solar_export_power has returned to available, the Reimann Sum sensor is unavailable until sunrise.

The downstream Reimann Sum sensors using the Grid Import sensor have no such issue, possibly because they are receiving changijng data?

I thought the "available" template command might help, but I am not sure how to use that.

BTW I thought this only started happening in May - ie: 2023.5.0

[erkr]

Hi,
As long the author doesn't recognise stable sensors as a valid use case, I created extra template sensors as a work around to add time triggers in case the value remains stable. See #96875 (comment)

[erkr]

@Steve2017 the other issue was related to loosing the source sensor when it shortly became unavailable. Maybe good to know that seems to be solved as well since I use the template workaround. Maybe related to your issue

[Steve2017]

since I use the template workaround.

It could be - but we shouldn't need a workaround.

I am trying to understand your solution because I still don't have a good understanding of the functions.

[erkr]

The integrate and derive integrations only trigger calculations when the source sensor updates. With the time pattern, the template sensor periodically changes the content of a dummy attribute, which also counts as an update, without changing the sensor value. So that triggers a new calculation for stable sensors

[AleXSR700]

It's not that it can't be done, it's just that it isn't a bug...

I missed this thread during my search and had opened a parallel issue here #102057

I also explained why it is a bug. Essentially, the current Rieman sum reports 0 kWh consumption even if the source constantly consumes 1 kWh. So the result of the integration is wrong.
If no update is performed, the returned value must not be 0.
Returning 0 is simply false. An hourly sum must either force an update even if the input data is constant or must return unknown/unavailable.

By the way, one could argue that the integration of a constant value is the simplest form of an integration ;-)

[Steve2017]

I have moved away from using this Riemann Sum sensor because it is unreliable. That rules out using the Enphase integration because it needs Riemann Sum for some data. I have moved to REST for my power/energy data and it is working without issue.

In my case the constant reporting (0kW) was not the problem. The system went off-line for 60 seconds and that caused the sensor to fail until it had a reading other than 0kW.

I found the comments that the issue with the Riemann Sum sensor ‘isn’t a bug but a feature request’, to be strange.

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[erkr]

bump

[ronweikamp]

I have made an attempt to solve this issue #110685

[erkr]

This looks exactly as the solution we need. Finally get rid of helpers. Thanks! Can't wait to have it into the release